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Preface 



How should you evaluate something like OIAS? Ultimately, 
in terms of its impact on the labor market performance of its 
users, but that, is expensive and time-consuming evaluation. 
There are also shorter-term criteria, most of which are easier 
to assess. Some have to do with conceptual and technical matters 
readability, validity, data sources, costs. Others have to do with 
the responses of users--can they operate the System, do they like 
it, do they think it is helpful? This report deals with the re- 
sponses of users. 

Here we have the accumulated experience and opinions of 
two hundred people- -conselors and clients--who used the System 
in actual career planning situations and under all the practical 
limitations of time, budgets, and competing demands. 

A word of explanation is due those who may wonder about 
the difference between OIAS and the Career Information System. 
Briefly, OIAS is an information delivery system; the Career 
Information System is an organization that (1) develops current 
occupational information; (2) manages occupational information 
delivery systems such as OIAS; and (3) helps Oregon schools and 
agencies integrate such information and delivery systems into 
their counseling and instructional programs. 

The report deals exclusively with OIAS usage in the Employ- 
ment Service. It was prepared by Jerry T. Weick, who was a 
counselor in the Concentrated Employment Program at the time 
of the Portland test, where he used the System with clients. The 
report thus reflects the perspective of a practicing counselor who 
tried using the System with some of the most difficult clients, 
the severely disadvantaged. 

Special thanks is due the counselors and counseling super- 
visors involved in the test for their cooperation, their extra effort 
on behalf of the experiment, and their constructive comments. 
Obviously the experiences and opinions of both counselors and 
clients vary, just as their needs, abilities and "styles" vary, but 
there are clear patterns of experience, and there is broad-based 
agreement on many important issues, 

Bruce McKinlay, Director of C IS 
University of Oregon 
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CHAPTER I 
SUMMARY 

This study establishes the effectiveness of Occupational 
Infornaation Access System (OlAS) usage in the counseling units of 
three Ennployraent Division offices in Portland, Two versions of 
OIAS, a conriputer- linked version and a nriaLiuai card-sort version, 
were connpared with the traditional mode where counselors deliver 
occupational information during the counseling process. OIAS was 
found to be a more effective mear.s of delivering occupational 
information for use in the ^career decision- making process. 

• Client self-use of OlAS was examined throughout the test. 
The experience of clients in completing the QUEST questionnaire, 
obtaining a list of occupational titles to explore and acquiring one 
or more specific occupational descriptions was monitored through- 
out the test, with counselor interpretation at strategic points. This, 
experience led most counselors to conclude that OIAS can be oper- 
ated by clients. Counselors judged that technical assistance and 
interpretation of the occupational lists are desirable. The vast 
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major ily of clients, both disadvantaged and non- disadvantaged, 
iound OlAS attractive, fim to use, and helpful in making job plans. 

Research findings based on the experience of 17 counselors 
and 267 clients, both disadvantaged and non-disadvantaged, include 
the following; 

--OlAvS usage enriched the occupational decision- making 
process by stimulating exploratory activity and introducing 
more order into the decision- making and counseling process. 
--OIAvS raises the level of client participation in the career 
planning process and facilitates client- counselor communi- 
cation. 

--Both the computer and card- sort versions effectively and 
efficiently assist the vast majority of clients in making job 
plans. 

--The computer version is more attractive than the needle- 
sort and preferred by most counselors, though both are 
effective delivery vehicles. 

--The information components of OIAS provide satisfactory 
and pertinent information in effective, yet varied formats. 
--OIAS made available more and better information within the 
time usually spent by counselors on information delivery. 
--There was some small savings in staff time. 



--With the exception of the severely disadvantaged who lack 
basic language skills, disadvantaged clients are able to use 
and gain the same benefits from System usage as non- 
d isadvantaged c lients , 

--There is a distinctly different occupational dec ision- making 
process between disadvantaged and non-disadvantaged clients. 
Overall the evaluation indicates that OlAS is an effective 
instrument for providing pertinent occupational information to the 
vast majority of the counseling clientele served by the State Employ- 
ment offices in Portland. No counselor indicated that the use of 
OIAS had any adverse effect upon any client. While the System was 
^expected to have its limitations, a number of especially helpful 
aspects are unanticipated benefits. The vast majority of clients 
indicated that they would use OIAS again if they needed information 
in the future and that they would like to see OIAS kept in the 
Employment offices where they had used it. 
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CHAPTER II 
BACKGROUND 

Occupational labor market information occupies an increas- 
ingly critical role in career choice and career decision- making 
processes. Increasing labor market complexity and change impose 
specialized and shifting demands upon the individual in the labor 
market and accentuate the need for individual career planning. 
One result of these changing conditions has been recognition of 
the need for new formats and methods for systematizing, process- 
ing, and delivering occupational information in ways which more 
adequately reflect these complex conditions and respond to the 
inforination needs of contemporary labor market participants. 

Response to this need resulted in the development of the 
Occupational Information Access System (OIAS) at the University 
of Oregon with funds and technical assistance provided by the U. S. 
Employment Service, Manpower Administration, U.S. Department 
of Labor. 

Stimulating the development of OIAS were some basic observa 
tions. P^irst, vocationaT planning and deci sion- making based on 
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reliable and valid information have a higher probability of success 
than plans nnade without such information. Second, while the U.S. 
Department of Labor has carried the primary responsibility for 
development of labor market data, new information is needed and 
much that is available is underutilized. The complex of agencies 
and private firms that offer this information and the reliance on 
printed media contribute at least partially to this underutilization. 
Third, the multiplicity of occupational classification systems makes 
comparable information accessible only to those knowledgeable 
about such classification systems. Consequently, most students 
and job seekers are unable and do not avail themselves of such 
.information, ^ 

The Occupational Information Access System 

OIAS is designed to make available information accessible and 
understandable by individuals making career plans. The person 
using the System typically begins by responding to the 25 questions 
in the QUEST questionnaire, typing his or her responses into the 
computer via a teletype terminal or using a manual needle-sort 

; ^Bruce McKinlay, "Occupational Information Access System: 
A Model System of Labor Market Information For Use In Coun- 
seling, " Journal of Educational Data Processing , Vol. 7, No. 5, 
p. I. 
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card file to obtain a list of occupational titles or a stack of occu- 
pational cards. From this point there are a number of directly 
avai lable information components, including cone ise occupational 
descriptions, a bibliography of the most important general and 
specific information sources about each occupation contained in 
the System, taped interview cassettes, and, depending upon 
particular system applications, an index of employers who hire 
persons in specific occupations or a file of persons working in a 
specific occupation who are available to discuss their occupation 
with an interested person. 

Summary of Research to Date 

A central thrust in the early development was the desire to 
build a delivery system which could be used directly by the person 
in need of occupational information, that is, as a tool for inde- 
pendent student or client use. This led to an early project study 
which assessed the readability of the QUEST questionnaire and the 
validity of self- report of the questionnaire as a means of gathering 
pertinent information. 

"Field testing of these materials showed very few 
readability problems. Over 90 percent of the counselors 
and clients in various schools and social agencies who 
tested the System rated it easy or very easy to use. In 
a special readability and validity test conducted in three 
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I'Jmploy mont. Service offices, wording problems were 
reported on only two percent of the QUEST c|uestions 
answered, and in only one half of one percent of the 
questions were wording problems associated with response 
error. Surprisingly, disadvantaged users reported fewer 
wording problems than non- disadvantaged users. 

Even with the more difficult occupational descrip- 
tions, 96 percent of the users (in this case high school 
students)..said they were easy or very easy to read. 

Readability does not seem to be much of a problem with 
Occupational Information Access System. The crucial issue 
has to do with the validity of self- reporting. 

In response to this question the report goes on to state the 

following. 

"There is some indication in this research that 
client self-reports are as valid as counselor estimates, 
and other research summarized in this report indicates 
that self-reporting is as valid as testing. Thus, while 
self-reporting may not be relied upon completely for all 
clients, it is at least one of the appropriate bases for 
questionnaire response. 

OIAS was pilot tested in six varied agency settings. These 
included a large high school in Eugene, two Oregon Employment 
Division offices, Lane Community College, a Vocational Rehabili- 
tation Division office and the University of Oregon Counseling Center. 



Bruce McKinlay, ""Validity and Readability of the Occupational 
Information Access System 'QUEST' Questionnaire," 1971, pp. 1-2. 



•^Ibid., pp. 2-3. 
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Major conclusions of the.' pilot testing indicated thai: the System 
was used extensively and constructively with very favorable results 
by a wide range of different kinds of students and agency clients; 
it adds to the thoroughness of career counseling; the teletype 
ternninal is a highly effective and attractive delivery device; the 
QUEST (questionnaire- list) process is effective; and the four 
information files received generally favorable response with the 
250-word descriptions receiving the greatest use.*^ This pilot 
test was most encouraging about the System^s applicability with 
students, but left unanswered questions concerning agency usage. 

On the basis of the positive results of the pilot testing in 
schools, two detailed evaluations were completed in 1971. A test 
of OlAS at the Counseling Center at I.ane Community College com- 
pared the delivery of occupational information by the Counseling 
Center and that of OIAS with ten characteristics of an ideal occupa- 
tional information delivery system. 

"Compared with a model occupational information 
checklist comprised of characteristics on which most 
counseling authorities agree, the connputer-based 
Occupational Information Access System (OIAS) rated 



"^Bruce McKinlay and Larry L. Ross, iLvaliiaiion of Occupation 
al Information Access System Use In Six Pilot Agencies , University 
of Oregon, Eugene, Oregon, 1970. 



"good" and the Lane Connmunity College Counseling 
Center "fair. "5 

More specifically the report noted that "all but one counselee, 
sought counseling out of a definite need for occupational informa- 
tion" and that "all but a few students were either very satisfied 
or satisfied with the information they received from either 
system, "^ 

"study results indicate that OIAS is at least as 
effective and definitely more efficient as an information 
delivery system. 

The ability of OIAS to deliver occupational informa- 
tion more efficiently and much less expensively than the 
Lane Community College Counseling Center does not mean 
that OIAS should be substituted for counselors. In fact, 
OIAS would be a poor substitute for a college counselor, 
since it orily serves the purpose of information delivery. 
Counselors, whose tasks typically involve personal 
advising as well as delivering occupational information, 
could benefit from using OIAS to obtain information. 
The time previously spent filing occupational materials 
and searching through innumerable information sources 
could be spent offering personal human advice to college 
students."' 



•^Larry Lynn Ross, The Effectiveness of Two Systems for 
Delivering Occupational Information: A Comparative Analysis , 
Master's Thesis, University of Oregon, Eugene, Oregon, 1971, 
pp. 81-82. 

^Ibid., pp, 81-82, 

'^Ibid., pp. 83-84. 
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A project report, published just after the community college 
study, evaluated usage of OIAS at Churchill High School iu Eugene 
where the System proved very attractive to the slightly more than 
1, 000 students who used it. Repeat usage was common and there 
was evidence that OIAS was not only interesting to students but 
actually helped them in career planning. Most indicated it gave 
them new ideas and those who used the System consistently showed 
that they knew more about job prospects on a test of knowledge 
than those who did not use it. About two-thirds of the students 
who used OIAS talked with their parrents about the System with 
most taking materials home with them. However, while OIAS was 
influential with students, it had little effect on established instruc- 
tional programs. Lastly, the study revealed through a comparative 
cost analysis that OIAS was only half as expensive as having either 
a teacher-aide or counselor man an occupational " information room,^ 

The Portland Employment Service Test 

With the results of the pilot testing and the encouraging out- 
comes of two major tests at Lane Community College and Churchill 

^Bruce McKinlay and Daniel Adams, Evaluation of the Occu - 
pational Information Access System As Used At Churchill High 
School , A Project Report, University of Oregon, Bureau of 
Governmental Research and Service, 1971, pp, 2-4, 
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High School, a full scale lest of OIAS in three Oregon State Employ- 
nrient Division offices was conducted to deternnine to what extent 
the sanne system would be applicable with agency clients. The 
results of this test constitute the present evaluation. The three 
offices were the Adult Opportunity Center, the Youth Opportunity 
Center and the Portland Concentrated Employnnent Progrann. 

In each setting the same test format was followed. Clients 
were selected from the current population of persons engaged in 
the process of making or changing an occupational choice through 
counseling. Each client selected was randomly assigned to one of 
three groups: a control group, an experimental group who used 
the needle- sort version, or an experimental group who used the 
computer version to access information. The test was conducted 
by the Employment Service counselors and their clients. Each coun- 
selor who participated worked with clients in all three test groups, 
thus insuring control for counselor differences. 

This report is based on evaluation data collected from three 
major sources. First, questionnaires were completed by clients 
in each experimental group. Secondly, a companion questionnaire 
was completed by each client's counselor. Both questionnaires 
included structured responses and unstructured comment sections. 
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Finally, OIAS project staff conducted follow-up interviews with each 
counselor who participated in the test. Thus, the structured re- 
sponses provide a framework which the unstructured observations, 
impressions, conclusions and suggestions elaborate and enhance. 
Together they present a balanced analytic base for the present 
evaluation.^ 

The major areas of emphasis in the evaluation are the im- 
pact of information on client occupational planning, the effectiveness 
of the QUEST questionnaire, the clients* and counselors' as.'.^^ss- 
ments of the content and format of the information, the extent of 
technical assistance needed by the client in using the System, and 
the effectiveness of OIAS for both non-disadvantaged and disadvan- 
taged clients. 

Throughout the test, counselors operated within their regular 
work schedules. No additional time was available for counselor 
participation in the study. Thus, counselors had to adapt and in- 
corporate the new procedure into their daily work schedules. While 
individual counselor interest varied from fair to enthusiastic, the 
overall level of cooperation was excellent. In spite of limitations, 
almost every counselor considered the test to be representative. 



^For detailed presentation of methodology see Appendix. 

ERLC 
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CHAPTER III 
IMPACT OF INFORMATION 

The impact of information on clients^ plans and activities is 
of paramount importance, since the reason for introducing occupa- 
tional information is to improve career plans. Counselors evalu- 
ated the effects of OIAS usage very shortly after a client finished 
using the System, so the reported outcomes are descriptive of 
expressed feelings, decisions and plans related by the client. 
The short term of the evaluation precludes follow-up examination 
of the clients' labor force behavior. In spite of this limitation on 
observed outcomes, the impact of information on client understanding 
and plans can be judged from counselor and client opinions at the 
time of the experiment. There| are several items in the evaluation 
that bear on this issue, and differences were pronounced between 
those who used the System and those who did not. 

Counselors were asked to rate, for each client, their overall 
impression of how influential the information had been. In three- 
fourths or more of the cases, counselors rated, information obtained 
through OIAS as influential. In this experiment there was a control 



group which received information from the counselor in the tradi- 
tional manner. The counselor presented the information to control 
. group clients when it seemed appropriate to him; whereas the 
experimental groups received information through use of OIAS as 
an identifiable step in the counseling process. 

Table 1 shows a relatively consistent pattern of counselor 
rating for all three .groups. The most striking difference is the 
rather high rating that counselors gave to information they them- 
selves presented to the control group. The descriptive comments 
of counselors give meaning to the difference. In cases where 
counselors said control group clients were not influenced by infor- 
mation given by the counselors, the counselor simply stated, "no 
effect, " "not influenced at all, " while they gave more explicit 
reasons, such as, "client is not ready to seek a vocational goal, " 

TABLE 1 

COUNSELORS' RATING OF INFLUENCE OF 
INFORMATION ON CLIENTS 

Experimental Groups Control 
Rating of Information Influence Computer Card- Sort Group 

Highly influential 18% 27% 40% 

Somewhat influential 62 48 51 

Not influential 19 23 8 

ERLC 
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and "client was referred from a halfway house, not ready for em- 
ployment yet/' when giving a low rating to information from OIAS. 
Since information presented through OIAS was presented as a dis- 
crete part of the counseling process, those results may have been 
viewed more objectively, though this is only speculation. 

More informative than ti ?se ratings are the responses of 
counselors when asked about identifiable actions taken by clients. 
The differences are pronounced. The behavior attributed to the use 
of information tended to be more specific for the OIAS experimental 
groups than for the control group. When talking about clients who 
used OIAS, counselors more frequently used words such as 'plans, " 
"planning," and "next step is...," and "began to plot definite steps 
toward action." When further counseling interviews were planned, 
the counselor more often expressed specifically the purpose of the 
additional interview;. Plans to- explore new or different occupational 
areas was reported far more frequently. These counselor comments 
reveal much more definite and precise plans and purpose for con- 
tinued counseling. 

In contrast, descriptions of control group client actions were 
more often couched in vague terms. Comments such as "he seems 
to have narrowed choice down. .., " "client became enthusiastic about 
continued counseling, " and "may try to find a job" were more 



frcqr.enl. When continued counseling was scheduled, the purpose of 
the additional interview tended to be expressed in less precise terms 
such as "to come back for further study." While counselors re- 
ported directing many control group clients to occupational informa- 
tion and resource materials, there was a general absence of 
comments that indicated any orderly exploratory process except 
where a specific occupation was the point of exploration. 

These are important and significant differences. Since assign- 
ment of clients to groups was random, and since each counselor 
participated in the test and evaluation of both control and experi- 
mental group clients, the differences cannot be attributed to client 
or counselor differences, but are a result of OIAS usage. As 
already discussed, since use of OIAS was a distinct component of 
the counseling process, the counselors were able to observe and 
specify outcomes more clearly. Thus, OIAS tended to operate as a 
feedback mechanism on the relevance of information to the client. 
It also tended to point out clearly those clients who were not ready 
to approach the occupational decision- making process and who 
needed counseling to help resolve more fundamental problems first. 
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Purpose of Information Usage 

The effect of OIAS is further elucidated by counselor ratings 
of the purpose of information usage by clients. Table 2 clearly 
points out that counselors see vocational exploration as the purpose 
of occupational information when OIAS is used, but they more often 
see decision- making as the purpose of information when information 
is delivered in the traditional manner. While a variety of inter- 
pretations can be offered, the comments of clients and counselors 
provide the best elaboration of these data. Although descriptive 
comments were not gathered from control group clients, the com- • 
ments of OIAS users are instructive. They commented on ''the 
diversity of job types and fields which (were) applicable, " "made 
me aware .pf some positions for which I was qualified that I had not 
considered," "it gave me more ideas," "provided some possible jobs 



TABLE 2 

COUNSELOR RATINGS OF HOW INFORMATION 
WAS USED BY CLIENTS 





Experimental Groups 




Purpose of Information 


Computer Card- Sort 


Control Group 


Exploration 


60% 60% 


41% 


Decision 


17 20 


.34 


Confirmat ion 


23 20 


25 
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I hadn't thought about before^' and "let me know of different job 
opportunities available that interested me. " These comments were 
typical of clients who sought counseling, and who were not seeking 
specific information or confirmation of a prior decision about a 
particular job. However, even in some of the latter cases, com- 
ments indicate changes in decisions based on information obtained 
from OIAS. It is apparent that OIAS adds a distinct and useful 
"exploration" phase to the career planning process. 

Efficiency of Information Delivery 

In the follow-up interviews most counselors indicated that, 
while OIAS did not reduce the amount of time spent in counseling . 
a client, it brought more information to bear and made more 
alternatives available within the same amount of time. Describing 
the impact of information obtained through OIAS, counselors made 
comments such as "it gave me a little something more to explore 
on, " "it took a shorter time to cover more, " "it broadened my 
information," "on many of my counselees it gave them other alter- 
natives; ideas that had not occurred to me," "a big help; helped 
organize where to go," "a lot. of information came "back on the com- 
puter (terminal) that they wouldn't have dug .out for . themselves, " 
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and "(OIAS) brought together itiformation it would have been iiard 
to accumulate. 

The preceding client and counselor comments provide explana- 
tion for the differences in counselor ratings between the control and 
experimental groups. OLAS usage provided more information and 
more alternatives within the same amount of time. Some coun- 
selors indicated that in some instances it would have required three 
or four limes as long to assemble, present, and utilize the same 
amount of information without the assistance of OIAS. A few coun- 
selors slated that they would never have been able to assemble and 
deliver the same amount of information without OIAS. Since more 
information and alternative choices were available to OIAS users, 
more expioralion resulted. 

Effect of OIAS Usage., on Decision- Making Process 

OIAS usage amplified the decision- making process, and, as 
indicated earlier, it tended to introduce more order into the occupa- 
tional decision- making and counseling processes. Counselors tended 
to take advantage of the increased information available through OIAS* 
This resulted in lengthening the decision- making process, because 
the additional information enhanced exploratory activity both quanti- 
tatively and qualitatively. OIAS usage also tended to clarify the 
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purpose of continued counseling. The differences in counselors' 
ratings of the purposes of information usage reflect this fact. 
The extent and net effect of this expanded decision- making process 
would require a longitudinal evaluation of continued counseling with 
the clients in both the control and experimental groups, but there 
is no question that the increased amount of information provided by 
OIAS enriched the occupational decision- making process. 



ERLC 
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CHAPTER IV 

EVALUATION OF CLIENT DEMAND FOR OlAS 
IN STATE EMPLOYMENT OFFICES 

Apart from the effectiveness of OIAS it is important to know 
whether clients will use it. Of course they can be directed and 
encouraged by counselors to use it, but one would hope for a 
system that clients would use as a matter of choice. In order to 
gain some estimate of the client demand for OIAS, they were asked 
two specific questions. 

First, they were asked whether they would use OIAS again if 
they needed job information sometime in the future. 
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TABLE 3 

CLIENTS WHO INDICATED THEY WOULD USE 
OIAS AGAIN IF THEY NEEDED 
JOB INFORMATION LATER 







Computer 


Card- Sort 


Total, All Clients 


96% 


94% 


88% 


Non- Disadvantaged 


92 


89 


94 


Disadvantaged'^- 


91 


98 


76 



'^'Clients of Employment Division offices and programs are 
classified as "disadvantaged" according to criteria defined 
by the U.S. Department of Labor and used to determine 
qualification of persons for special programs and services. 
The criteria include racial minority membership, lack of 
a high school diploma or G.E.D., whether the person is 
handicapped, younger than 22 or older than 45 years of 
age, and prior annual income below the federally- defined 
poverty level. A combination of these factors make a 
person eligible for program services. 

These ratings reveal an extremely positive overall response to 
OIAS and a very definite future demand for the System. There 
were instances of clients leaving some Employment Service offices 
to go to the Youth Opportunity Center (YOG) or the Adult Opportunity 
Center (AOC) "where they have all that information on the computer." 
It is particularly significant that 98% of the disadvantaged clients 
would use the computer version of OIAS again, while disadvantaged 
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clients who used the card- sort showed appreciably less enthusiasm. 
Still, the 76 percent positive response constitutes definite support. 
The difference between ratings of the non-disadvantaged experimental 
groups cannot be considered significant. The pattern of responses 
in the four categories are consistent with the conclusions of the 
relative effectiveness and attractiveness of the computer and card- 
sort versions for disadvantaged clients. 

The other pertinent question asked of clients was whether 
OIAS should be kept at the Employment Division. Table 4 .'shows 
the responses of experimental groups to this question. 

TABLE 4 

" Ci;iENTS' RECOMMENDATION ABOUT KEEPING 
OIAS AT THE EMPLOYMENT DIVISION 

Non- 

Disadvantaged Disadvan- Computer Card- Sort 
Clients taged Clients Version Version 



(92% of Clients Responded) 



Definitely Yes 


60% 


49% 


60% 


47% 


Yes 


39 


49 


39 


49 


No 


1 


2 


1 


4 


Definitely Not 


0 


0 


0 


0 



o 

ERIC 



24 



This question shows the same strong pattern of demand among 
clients. Again it is significant to note the very high proportion of 
disadvantaged who responded with strongly positive ratings. Evi- 
dently some clients who indicated they would not use it again or 
who felt they were not helped by the System themselves, thought 
that others would benefit from using it. From these results, there 
appears to be a definite demand for OIAS in the Employment Divi- 
sion offices in Portland, Client responses corroborate the need 
expressed by counselors for OIAS an an information resource, and 
suggest that it would have continued usage. 
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CHAPTER V 

TYPES OF CLIENTS WHO CAN BENEFIT FROM OIAS 

Evaluation indicates that OIAS is an effective instrument for 
providing pertinent occupational information to the vast majority 
of the counseling clientele served by the State Employment offices 
in Portland. This includes the disadvantaged as well as the non- 
disadvantaged population. However, it is possible to be somewhat 
more precise about the range of effectiveness of the System, This 
can be determined from analysis of clients for whom OIAS is par- 
ticularly helpful and efficacious and those who are little helped or 
who cannot benefit from using such a system. 

To assist in this evaluation, counselors were asked if OIAS 
is more useful with one kind of client than another; their comments 
are diverse. Four counselors specifically mentioned the severely 
disadvantaged, or non-readers, as people for whom OIAS was not 
especially helpful, but one counselor said ine System was particu- 
larly appealing to the non-reader. Five counselors mentioned that 
the System was more useful with clients who were committed to 
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finding a direction and choosing an occupational goal; while two 
counselors reported that OIAS was particularly helpful in drawing 
out passive or withdrawn clients and in resolving impasses in the 
counseling process. 

Counselor comments are more consistent regarding the limi- 
tations of OIAS. When asked if there are particular clients with 
whom OIAS is bad, one-third of the ' counselors reported there were 
no clients for whom the System was inappropriate. Others said 
that clients with low abilities, the severely disadvantaged or clients 
with very low reading skills are the persons helped least by OIAS. 
Finally, one counselor said the results of QUEST were not suffi- 
ciently discriminating for handicapped persons. Evidently, certain 
limitations resulting from handicaps are not reflected in responses 
in the questionnaire, so some inappropriate occupations appear on 
the list. 

In the variety of comments there are some consistent themes. 
First, OIAS is not well adapted to the severely disadvantaged per- 
sons with very limited or no reading skills, though the disadvantaged 
in general benefited substantially. Secondly, it appears that persons 
with very low abilities tend to be discouraged by OIAS, and probably 
need extra counseling to make a sound and satisfactory occupational 
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decision. Thirdly, even the attractiveness of the process cannot 
motivate the client who is not really interested in an occupational 
goal. 

None of these three linnitations is a nnajor defect of the 
System. Rather, they help delineate the areas of the System's 
effectiveness, and provide guidelines as to which clients should 
use OlAS. 

It is significant that no counselor indicated the use of OIAS 
had any adverse effect upon any client. While the System is 
expected to have its limitations, a number of the especially helpful 
aspects are unanticipated benefits. In addition to providing the 
counselor valuable information, usage of OIAS can assist the coun- 
seling process to facilitate counselor-client communication. It offers 
an alternative means of expression of interests, preferences, and 
skills for clients who have difficulty verbalizing these matters to a 
counselor. The objective, character 6f the System provides both 
counselor and client with an expanded base of . information about the 
client as well as relevant occupational information. Its attractiveness 
for clients is another positive feature which increases the counselor's 
credibility with clients and raises the level of client participation in 
the career planning process. 

ERLC 
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CHAPTER VI 

ANALYSIS OF USAGE OF QUEST 
QUESTIONNAIRE AND LIST 

The QUEST questionnaire is a key component of OIAS, since 
it furnishes the introduction and starting point for the typical System 
user. The ease of usage, the efficiency and effectiveness of an 
introductory component such as QUEST, contributes substantially to 
the value and effectiveness of the entire system. Therefore, a wide 
range of factors were considered and included in the evaluation, A 
range of structured responses and descriptive comments about the 
process, content and format of the questionnaire and resulting list 
of occupational titles were elicited from both clients and counselors. 

Client Self-Use 

Self-use was a basic procedural feature utilized in testing 
QUEST and was a specific point of interest in the evaluation. The 
efficacy of self-reporting on the questionnaire depends upon its 
readability, comprehensibility and the clarity of the printed instruc- 
tions, as well as upon the validity of self-reporting generally. 
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A separate study of the technical issues of readability and validity 
of the QUEST questionnaire has been done. Of interest here are 
the operational considerations of client self-use. 

Though nnuch data relating to client self-use was collected, 
the best conclusion comes from the follow-up interviews, when 
counselors were askeu how they would use QUEST if they were free 
to use the questionnaire any way they wanted to in their counseling. 
This question covers the four major steps involved in using the 
questionnaire. (See Table 5.) 

The overall pattern of responses indicate that counselors 
largely see QUEST as a tool for client use. Only one counselor 
on one item, a mechanical procedure, thought that the counselor 
should perform the principal role in that activity. 

Filling Out the Questionnaire 

Seventy percent of the counselors thought the client capable of, 
completing the questionnaire alone or with only routine technical 
assistance. Among the 30 percent who. thought counseling and inter- 
pretation necessary at this step were counselors who expressed 
doubts about client ability to self-report, and who preferred to inter- 
pret and restate questions in an attempt to correct any distortions 
in client judgment. The distribution of counselor ratings on filling . 
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out the questionnaire reflects differences in individual counseling 
styles as well as client abilities; to this extent the ratings cannot 
be construed as pure judgments of client capability to fill out the 
questionnaire. 

Readability and comprehensibility of the questionnaire and 
instructions were important aspects of the self-use format. When 
asked to assess the readability of the questionnaire for clients, 69 
percent of the counselors indicated that there was no problem with 
reading for most clients, although there was ambiguity in some 
specific questions. One counselor who worked at the Concentrated 
Employment Program commented that reading difficulty for disadvan- 
taged persons was generic, and not a problem of the questionnaire 
or the System. 

During the test, counselors were asked to review the client^s 
questionnaire with him before getting a list. In the final interview, 
counselors were asked about the necessity of this procedure. While 
77 percent thought some review was either necessary or preferable 
with most clients, most counselors indicated that the review was 
brief and principally aimed at spotting obvious errors or inconsis- 
tencies. 
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Reading and Understanding the Ins t ructions and Using the Terminal 

The vast nnajority of counselors thought that clients were 
capable of reading and understanding the instructions, typing the 
answers into the terminal and sorting the cards. However, they 
saw the availability of routine technical assistance during these 
steps as necessary. 

Descriptive comments of clients and counselors reveal some 
pi'-oblexns with mechanical malfunctions of the terminal, but most 
difficulties stemmed from minor procedural errors of clients in 
using the terminal. These difficulties generally resulted from a 
failure to follow the instructions properly, or from* client^s un- 
familiarity with any similar kind of machine operation. Most of the 
dilticulties experienced at this stage were easily overcome with minor 
technical assistance, but some clients needed considerable assistance 
to complete this step. Even in these cases it was still a matter of 
providing technical assistance rather than counseling. No counselor 
commented that the instructions were difficult to read or understand. 
Rather, they tended to comment on client ability to understand and 
apply the instructions. In fact, one counselor noted that client ^?.cti~ 
vity while using Oli^S provided a behavior sampling which indicated 
client ability to follow instructions and operate equipment. 
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Reviewing the List of Occupational Titles 

Reviewing the list of occupational titles is the step where re- 
porting has been completed and client decision- making becomes the 
major activity. After obtaining a list, the format of OIAS becomes 
much less structured and allows the client a number of different 
options. The alternatives include seeking specific job descriptions 
and occupational information, determining why a particular occupation 
was not included in the list, changing responses to specific questions 
on the questionnaire, and discussing the resulting list in order to 
insure that the client understood the meaning of the results and to 
rectify any errors which might have developed in the previous steps. 

There were three basic questions regarding client self-use of 
- QUESTT— (l-)-Gan-they self-report abilities and preferences? . ' - 
(2) Can they understand and operate the System technically? (3) Can 
they adequately interpret the results and prepare their own career 
plans? 

The question of self-reporting was the subject of a project 
technical paper, "Readability and Validity of the QUEST Question- 
naire. " This test adds the opinions of practicing counselors who 
used the System. Most of them hold the opinion that self-reporting 
is a reasonable first approximation, and that brief review of the 
completed questionnaire and the opportunity for later revision 
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provide adequate means for resolving any problenns clients have 
with self-reporting. 

Secondly, this test of OIAS attempted to discover whether cli- 
ents could operate the System independently. This was a particu- 
larly essential issue regarding the use of QUEST. Results indicate 
clients are willing, most are able and many prefer to operate the 
System themselves, with occasional technical assistance required. 
The Portland test has shown that it is possible to design an occupa- 
tional information system which clients can operate technically. 

The third issue, inHependent client interpretations, was not 
tested, rather, counselors were required by test procedures to 
evaluate and interpret results. They believe counselor interpreta- 
tion of results to be the appropriate. place for counselor partici- 
pation, and recommend that as regular procedure. 

Counselor Time Required 

Time estimated by counselors to gather comparable pertinent 
information must be distinguished from the total time spent by 
counselors with clients in using OIAS. During the test, counselors 
provided all the technical assistance to their clients and in many 
cases spent time observing their clients* use of the System. This 
was included by counselors as time spent. When counselors reported 

ERIC 
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that OIAS didn't save time but provided more information in the 
same amount of time, they were including the time they spent pro- 
viding technical assistance as well as time spent observing the 
process. 

The time reports of counselors show that they spent the same 
time with the System as their clients- -about 40 minutes. As will 
be seen shortly, the reason lies in the test procedures, not in the 
demands of the System. Counselors were instructed to intervene 

at two places in the QUEST process, and were required to report 

\ - 

additional information about client use of the information files. 

Thus, most counselors found it expedient to stay with their clients 

while they used the system, to counsel, assist, or observe. 

Cpunselors do not need to assist and observe, however. Since 

routine technical assistance can be provided by persons other than 

counselors, it is appropriate to limit comparisons to the amount of 

counselors' time required. 

Reviewing the completed questionnaire with the client before 

using the terminal, and discussing the client's resulting list of 

occupational titles with him were the two steps where counselors 

thought some counseling and interpretation were necessary. For 

the first step, counselors reported that they spent a median time 
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of ten minutes with both non-disadvantaged and disadvantaged clients. 
For the later step, they spent a median time of two minutes with 
non-disadvantaged and five minutes with disadvantaged clients. From 
the brevity of time spent in both steps, it can be concluded that in 
a majority of cases counselors were mainly checking for obvious 
errors and misunderstanding of questions and screening for incon- 
sistencies in the resultant list of occupational titles. Development 
of a more efficient means of providing technical assistance would 
significantly reduce the time spent by counselors observing and 
assisting clients using OIAS. 

Readability 

While counselors reported some reading difficulties for a 
portion of disadvantaged clients, few such clients reported that 
they had difficulty. The following table reflects client responses 
to specific aspects of questionnaire usage. 

Table 6 shows that clients consistently rated readability very 
high. There is little difference between the ratings of non-disadvan- 
taged and disadvantaged clients. What difference there is follows the 
expected pattern. The proportion of clients rating the questionnaire 
easy to use is significantly higher than the proportion of counselors 
who assessed the questionnaire as presenting little or no problem of 
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TABLE 6 

CLIENT RESPONSES ON EVALUATION 
OF QUEST QUESTIONNAIRE 



Non-Disadvantaged Disadvantaged 
Clients Clients 





YES 


NO 


YES 


NO 


Easy to Read 


96% 


4% 


94% 


6% 


Fun to Use 


90 


10 


86 


14 


Asked the Right Questions 


80 


. 20 


84 


16 


Related Jobs to My Own 
Likes and Dislikes, 
Values and Skills 


90 


10 


86 


14 



readability. However, counselors more frequently focused attention 
on technical considerations such as ambiguity and the adequacy of 
specific questions on the questionnaire. Thua, counselor assess- 
ments of readability are more comparable to client responses to the 
question whether QUEST asked the right questions. Ambiguity in 
certain questions and insufficient questions to delineate certain re- 
sponse categories were the most frequently cited weaknesses. In 
spite of these weaknesses and need for some revision, the vast 
majority of both non~disadvantaged and disadvantaged clients found 
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QUEST enjoyable to use and the resultant list of occupations 
relevant to their own abilities and preferences, 

QUEST List 

The list of occupational titles produced by the QUEST was 
another focal point in the evaluation. The length of lists, client 
perception of the relationship between the questionnaire and list, 
and the extent to which the list provided new ideas, new information, 
and new alternatives were central factors. Responses were elicited 
from both clients and counselors on these aspects. 

Though most lists were of useful lengths, a number of clients 
and counselors stated that particular lists were too long, while others 
were too short. Approximately half the counselors reported either 
that there was no problem with the length of the lists or that any 
confusion created by a long list could be resolved through counseling 
and interpretation. When a particular client received a list which 
seemed too long or too short, or when the client perceived the list 
as incongruous with his or her interests, clients often expressed the 
desire to fill out a second questionnaire, or to go back and be more 
specific on some answers. Examples of such instances reported by 
counselors included, "Client found that at the end of the first , card- 
sort he needed to be more specific on the questionnaire. Therefore, 
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he went through it again;" "We did QUEST twice over two sessions 
as client had a very long list of occupations, i Counselee felt he 
needed to think about it more carefully and do it again;" "Changed 
responses on questionnaire to narrow remaining choices;" and "The 
counselee wanted to and did go through two questionnaires to get 
better reflection of the possibilities for him. This seemed valid to 
the counselor as his final list with changes was actually more 
realistic for him. " 

These descriptive comments are representative of the use made 
of QUEST by most clients. They indicate that QUEST was creatively 
manipulated as a tool by the client rather than used in a rigidly 
mechanical fashion. These results allayed concern that clients 
might treat the results in a dogmatic and inflexible manner. They 
are also evidence that clients generally understood QUEST in the 
context of the occupational decision- making process. As one coun- 
selor stated, "Most clients use the list as a jumping off point." 

Counselors were asked directly whether clients were inclined 
to take the lists too seriously, and whether the computer version 
posed a greater danger in this regard than the card- sort version. 
Eighty percent of the counselors thought that there was no danger of 
the client taking the list too. seriously. But half of these counselors 
preferred that there be explanation or interpretation to insure that 

O 
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the list was understood properly. About 20 percent of the coun- 
selors thought there was more danger with the computer version; 
however, none indicated whether counseling and interpretation would 
relieve this danger. Thus, they were judging the danger in the con- 
text of self-use by client alone. Additional counselor ratings and 
responses confirmed that the majority of counselors thought it was 
at least desirable, if not essential, for the counselor to be involved 
with the client in discussing his list, though mostly for other reasons. 

There is no evidence in the responses of clients to indicate 
that the list was restrictive or taken too seriously. When asked to 
comment on the list, clients either remarked about the new occupa- 
tions it contained, or suggested more, specific questions. 

Value of List 

The amount of information provided by the questionnaire and 
list was an important consideration in assessing their value. When 
asked if the questionnaire and list gave them some new job titles to 
consider, clients' responses show that it was highly productive of 
new ideas. Table 7 summarizes these responses. 

The vast majority of both non~disadvantaged and disadvantaged 
clients indicated they received new alternatives and possible new 
directions through use of QUEST. Ninety-one percent of the 

ERLC 
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TABLE 7 

CLIENT RESPONSES WHEN ASKED IF THE 
QUESTIONNAIRE GAVE THEM SOME NEW 
JOB TITLES TO CONSIDER 
FOR FUTURE WORK 





Non- Disadvantaged 
Clients 


Disadvantaged 
Clients 


Definitely Yes 


32% 


22% 


Yes 


59 


54 


No 


8 


23 


Definitely Not 


1 


1 



non-disadvantaged clients reported this fact; the proportion of posi- 
tive responses of disadvantaged clients was slightly lower, reflecting 
some real differences between the two groups which are elaborated 
later in the report. When asked what part of the System was most 
helpful, both non-disadvantaged and disadvantaged clients rated the 
questionnaire second only to the job descriptions. Descriptive com- 
ments of clients frequently emphasized the helpfulness of the informa 
tion.. Of special interest was the fact that next in frequency to 
comments on the value of the information, disadvantaged clients 
reported that it was easy to use, fast, and efficient. 
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The QUEST lists, though sometimes long, were useful in 
encouraging occupational exploration. As will be seen in the next 
section, theyaccomplished their principal purpose of stimulating 
use of occupational information sources. But they did more. In a 
large percentage of cases they made people aware of potential 
career fields they had neither known nor considered before. 

Counselor Evaluation of Each Item in 
QUEST Questionnaire 

Detailed analysis of the QUEST questionnaire was a central 
objective of the follow-up interviews with counselors. Comments of 
counselors were sought on the questionnaire as a totality as well as 
on each question contained in it. Emphasis was placed on encour- 
aging the counselor to express his or her suggestions for modifying 
and improving questions. 

Counselors were generally satisfied with both the content and 
wording of QUEST questions. A large majority usually favored 
keeping the questions as worded. None was found to be grossly 
ambiguous, misleading or irrelevant, though some posed more prob- 
lems than others. Only the vision and education questions elicited 
a preponderance of suggestions for change in wording. The sex and 
region questions were the only questions any counselors thought should 
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be dropped. No counselor indicated that the number of questions 
was too large, while six counselors suggested that nnore questions 
should be added either to include or expand upon such factors as 
interests, teniperannents, and personality characteristics. Two 
counselors thought it would be helpful if GATB scores could be in- 
cluded, and one counselor suggested devising a nieans to include* 
results of the Interest Checklist. 

Counselor ratings and comnnents on individual questions follow. 
Only negative connments are listed. If counselors indicated satis- 
faction with a particular question, they were not asked to report the 
reasoning for their affirmative judgment. 

Keep 

as is Change Drop 
Physical Limitations : * 

1. LIFTING Could you do medium or 
heavy work? Thai: means a job 
where you lift 50 pounds or often 
lift and carry 25-pound objects. 
It also means lots of walking, 
standing, stooping, reaching, or . 
moving things, (Responses: Yes; 
No; Could, but wouldn't want to; 

Don't Know) 14 3 0 

Comments: Counselors who suggested change thought there was 
some ambiguity in the question. One reported that some clients 
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Keep 

as is Change Drop 

didn't understand whether to answer the question in terms of 
ability or willingness to take a job involving lifting. 

2. IMPAIRED VISION Do you have 
very bad eyesight? That is, are 
you unable to do close work; are 
you unable to tell distance, or are 
you color blind? (Responses: 

Yes; No) ^ jq ^ 

Comments: Counselors accepted the relevance of the question, 
but some pointed out that the question was ambiguous for persons 
who wore glasses or who have defective color vision only. Re- 
wording has since corrected the problem with regard to glasses. 

3. DEAFNESS Are you deaf or un- 
able to speak? (Responses: Yes; 

No) 12 5 0 

Comments: Counselors who saw a need for change in the ques- 
tion indicated that no provision was made for varying degrees of 
' partial hearing loss. However, since the Worker Traits do not 
provide information on degrees of hearing required, such re- 
wording is impossible. 
Location - - 

4. REGION Where are you willing 

to live? (Responses: Lane County 
Area; Pacific Northwest; anywhere 

in the U.S.) 14 2 1 
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Keep 

as is Change Drop 
Comments: Most counselors were satisfied with the question. 
One said the question should specify the city where the terminal 
or card- sort is located, and one desired a further breakdown of 
:a large metropolitan area, e.g, Portland, into several districts 
within the metropolitan area. This is infeasible with present 
data sources. The counselor who suggested dropping the ques- 
tion said it was unnecessary, 

CITY SIZE Some people want to 
work only in a large city; others 
want to work in a small city; 
others want to v/ork in a small 
town. What size community are 
you willing to work in? (Responses: 
Large city; Large city or small 
city; Small city; Small city, small 
town, or rural area; Rural area; 

No preference) 15 , 2 0 

Comments: The same counselor gave the same suggestions for 
further breakdown of large metropolitan areas. The other coun- 
selor suggesting change indicated that some people had difficulty 
making up their minds on how to answer the question. The vast 
majority favored the question as worded, and none wanted it 
deleted. 



WORKING CONDITIONS What kind 
of jobs would you take? (Responses: 
Indoors; Outdoors; No preference) 15 



2 



0 
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Keep 

as is Change Drop 
Comments: Both counselors seeing a need for change said the 
item should include category of combined indoor and outdoor 
conditioPiS. All wanted the question kept. 

SEX Some jobs (such as barber) 
are mainly filled by men; others 
(such as nurse) are mainly filled 
by women. I want a job mainly 
filled by: (Responses: Men: 

Women; No preference) 13 13 

Comments: Counselors who judged it desirable to retain the 
question referred to the fact that from a counseling standpoint 
it brought out attitudes and hang-ups of clients, it made them 
think about significant situations, and brought the question out 
for discussion. The counselor who suggested change indicated 
uncertainty whether to keep it or drop it. He pointed out that 
some jobs traditionally filled by women, such as waitress and 
secretary, in fact may be jobs working primarily with men. 
Counselors reported that most clients checked question as "no 
preference. " Of the three counselors who suggested dropping 
the question, one indicated no strong feeling about the question 
and the other two indicated the risk of stereotypes. 
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Keep 

as is Change Drop 

Education and Training 

8. EDUCATION How much training 
would you be willing to get? 
(Responses: No special education; 
High school; One year; Two years; 

Four years; Graduate) 4 13 0 

Comments: The vast majority of counselor criticism of this 
question indicated that it failed to distinguish between current 
achieved education level and willingness to pursue further educa- 
tion. Since many clients who completed the questionnaire .were 
interested in the possibility of MDTA training, there was a 
marked tendency for the client to indicate interest in obtaining 
as much training as possible. It further failed to distinguish 
how much further education a person was able to attain based 
on his own financial resources. Note, however, that they all 



thought it important to keep the question. 
Apt itudes 

9. WORKING WITH HANDS How good 
are you at doing fine work with 
your fingers and hands? Can you 
handle small things easily, quickly, 
and accurately? (finger dexterity) 
(Responses: ' Very good; Fairly 

good; Fairly poor; Don't know) 17 0 0 

10. EYE FOR ACCURACY How good 
is your eye for detail? Are you 
good at. finding errors in words 
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Keep 

as is Change Drop 

Aptitudes (Cont. ) 

or numbers, or at copying material 
accurately? (clerical aptitude) 
(Responses: Very good; Fairly good; 

Fairly poor; Don't know) 16 10 

11. ABILITY WITH WORDS How good 
are you with words? Do you 
usually understand the meaning of 
words? Can you express your- 
self well in speaking and writing? 
(verbal aptitude) (Responses: 
Very good; Fairly good; Fairly 

poor; Don't know) 15 2 0 

12. ABILITY WITH NUMBERS Are 
you good with numbers? . Can you 
do arithmetic problems quickly 
and correctly? (numerical 
aptitude) (ResporxScs: Very 
good; Fairly good; Fairly poor; 

Don't know) 16 " 1 0 

13. CATCHING ON TO THINGS Do 
you catch on to things easily? 
Are you good at understanding 
instructions and the reasoning 
behind thenn? Are you good 

at figuring out things? (general 
learning ability) (Responses: 
Very good; Fairly good; 

Fairly poor; Don't know) 17 0 0 

Conaments: There was general satisfaction with these aptitude 
questions. The two counselors who saw a need for change of 
two questions indicated reasons such as, people are too subjec- 
tive, too likely to read in their own feelings about themselves, 
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Keep 

as is Change Drop 
and that the questions failed to delineate two of the abilities 
adequately. 
Interests 

Working with Things 

14. PRECISION WORK Would you like to 
adjust and repair equipment or use 
instruments or tools to do precision 
work? Work in mechanics, metal 
working, office machinery repair, 
drafting, laboratory testing, the 
building trades, and sewing are 
precision work. (Responses: 

Yes; No; No preference) 15 2 0 

15. OPERATING EQUIPMENT Would 
you like to operate machinery? 
This could mean running a wood-, 
working machine, a computer, or 
office equipment; or it could mean 
driving equipment such as buses 
or bulldozers. (Responses: 

Yes; No; No preference) 12 5 0 

16. HANDLING MATERIALS Would 
you like handling materials? You 
might start and stop machines, 
steadily remove materials from 

a machine, sort small parts, or 
move things from one place to 
another as in production work, 
logging, farm work, sorting and 
grading, assembly work, and 
stock work. (Responses: 

Yes; No; No preference) 12 5 0 
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as is Change Drop 

Working with Information 

17. INTERPRETING FACTS Would you 
like figuring out new ways of doing 
things, keeping track of a lot of 
things at once, analyzing information, 
and determining what to do? This 
includes jobs in management, medi- 
cine, engineering, teaching, law, 
repair work, etc. (Responses: 

Yes; No; No preference) 15 2 0 



18. ORGANIZING AND USING 
INFORMATION Would you likei 
to gather information and put it 
together^ and then know what to 
do with the information? People 
in office work, laboratory work, 
and in the building trades use 
information in this way. (Responses: 
Yes; No; No preference) 15 

19. COPYING, SORTING, AND 
PUTTING THINGS TOGETHER 



Would you like to copy or type, 

to follow step- by- step plans to 

make things? Would you like to 

compare and . sort things? If so, 

you might consider jobs where 

you use. information in this way. 

(Responses: Yes; No; No 

preference) 13 



Comments: Criticism of questions 14 through 19 by counselors 
v/as aimed at the same common themes. First, it was reported 
that it was difficult for clients to distinguish between questions 
within these two groupings. Second, the work examples posed 



51 

Keep 

as is Change Drop 

problems, with some questions needing more examples while 
others seemed to limit consideration because the examples were 
limiting. Some counselors thought there should be more deline- 
ation between questions. Questions 15 and 16 were most often 
cited as needing more delineation. Questions 17* through 19 
tended to be judged as particularly difficult for clients to under- 
stand and answer. 
Working with People 

20. COUNSELING Would you like to be 
responsible for helping one person 
at a time find a solution to his 
personal, legal, medical or other 
problems? (Responses: Yes; 

No; No preference) 13 4 0 

21. DISCUSSING, BARGAINING Would 
you like to exchange information 
and opinions or bargain with others 
to reach decisions about program, 
policies, and actions to be taken? 
IVIanagers, health inspectors, and 
union bargaining agents, for ex- 
ample, spend a lot of time dis- 
cussing and bargaining. (Re- 
sponses: Yes; No; No preference). 12 5 0 

22. TEACHING, SUPERVISING Would 
you like to teach things to people, 
to explain work; procedures and to 
assign work to people? Would you 
like to keep up good working rela- 
tions among workers and judge their 
work? Supervision, teaching, and 
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Keep 
as is 



Change 



Drop 



recreation are examples of this 
kind; of work. (Responses: 
Yes; No; No preference) 



14 



3 



0 



23. 



SELLING, PERSUADING Would 
you like to sell or try to talk 
people into something? 
(Responses: Yes; No; No 
preference) 



14 



3 



0 



24. ASSISTING Would you like a 
job where you have to be 
pleasant to many different 
customers or other workers 
while you give or get instruc- 
tions? Waitress, sales clerk- 
ing, stewardess, library, and 
reception work are examples 
of assisting jobs. (Responses: 

Yes; No; No preference) 15 2 0 

Comments: Counselors who suggested chciages on items 20 through 
24 reported difficulties with semantics and difficulties of clients 
distinguishing between these questions. Several counselors indi- 
cated that the word "counseling^' was loaded and clients tended 
generally to want to help people so they responded positively to 
this item. 



5. MONTHLY WAGE How much must 
you make (for full-^time work) 
before you would consider working 
in a job field? (Responses:. 
Minimum wage; $350/month; 



Earnings 



$500/ month; $700/ month; 
$1, 000/month; Not important 
at this time) 



Keep 
as is 
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Change Drop 



0 



Comments: The six counselors who said the question needed to 
be changed had three specific suggestions. They saw the need 
for more categories of rate levels, the need to state, explicitly 
the band of wages in each category^ and the need to distinguish 
between the rate of pay at which a person is willing" to start and 
the final wage rate or salary, on a career basis. All favored 
keeping the question. 
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CHAPTER VII 

EVALUATION OF INFORMATION 
CONTENT AND FORMAT 

Process and content are complennentary aspects of information 
presentation. The impact of information on users of OIAS is a 
function of the process and format^'of information presentation as 
well as the actual content of the information. This is an extremely 
important fact which is generally ignored, but which has been 
brought home time and again during this project. In the previous 
section, evaluation of client use of QUEST showed that clients 
learned about the process of occupational decision- making from the 
questionnaire even though its ostensible purpose was merely report- 
ing. Information process and content are obviously intertwined. 

The QUEST list provides the typical starting point for users in 
obtaining detailed and specific occupational information. At that 
point in the process the user has four ways of obtaining specific 
information. Printouts of 300-word occupational descriptions are 
availablp for each occupation through the teletype computer terminal. 
An indexed bibliography and books provide additional detail. 
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Cassette tapes of interviews with people in a liniited number of 
occupations were available for client use, providing innpressionistic 
information about specific occupations. Lastly, an employer index 
shows the names of local employers and the number of persons 
working for the employer in given occupations. 

At this point it is appropriate to present the proportional usage 
of the components of OIAS. The pattern of usage provides a^ com- 
parison of information sources used by control and experimental 
group clients. Table 8 shows the frequency of usage of the specific 
information components of OIAS reported by clients in the experi- 
mental groups. 

TABLE 8 

FREQUENCY OF CLIENT USE OF 
SPECIFIC INFORMATION 
COMPONENTS OF OIAS 



Component 


FREQUENCY 


OF USE 


N on- Disadvantaged 
Clients 


Disadvantaged 
Clients 


QUEST 


100% 


100% 


Job Descriptions 


95 


88 


Bibliography and Books 


29 


11 


Employer Index 


11 


9 


Cassette Tapes 


6 


7 
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All clients who participa ad in experimental groups used the 
questionnaire. Table 8 reflects the proportion of those who used 
QUEST and went on to use one or more components of the System. 
Non~disadvantaged persons were more likely to use multiple informa- 
tion components than were disadvantaged clients. The most definite 
difference was use of the Bibliography and Books. This difference 
was expected, since this component depended heavily on clients 
looking up and reading printed information. 

Descriptions 

The occupational descriptions were the most frequently and most 
widely uaed specific information component. They also provided the 
most easily obtainable information. Typing "DESC" and an occupa- 
tional code number into the terminal provided a printout of a concise 
description of the nature, the requirements, the characteristics and 
employment prospects of that occupation. Evaluation of the content 
and format of the job descriptions is based on specific ratings and 
descriptive comments of clients and counselors. When clients were 
asked to rate which component of the System, including the question- 
naire, was most helpful, 60 percent of the non-disadvantaged clients 
and 52 percent of the disadvantaged clients rated the job descriptions 
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as most helpful. The following table shows how clients rated a 
number of characteristics of the job descriptions. 

Obviously, the client is not technically qualified to assess the 
characteristics of completeness and accuracy of the information. 
Nevertheless, their perceptions are crucial and asking clients to 
report their perceptions of the information indicates the way in which 
technically correct information will be viewed by clients. 



TABLE 9 

CLIENT EVALUATION OF JOB DESCRIPTIONS 





Non- 


Disadvantaged 




Disadvantaged 






Clients 




Clients 




Yes 


No Don't Know 


Yes 


No Don't Know 


Accurate and 










Up to Date 


92% 


5% 3% 


80% 


2% 18% 


Complete 


81 


11 8 


81 


9 10 


Easy to Read 


99 


1 


93 


7 


Fun to Use 


97 


3 


95 


5 


Related the Job 










to My Own 










Likes and Dis- 










likes, Values 










and Skills 


87 


13 


96 


4 
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'Pho fad iluil clients found the descriptions easy to read and fun 
to use, as weLL as easy to obtain, contributed to their popular and 
widespread usage. These factors coupled with the fact that most 
clients perceived them as complete, accurate, and personally rele- 
vant are fully consistent with clients' rating the job descriptions as 
the most helpful specific information component. The large number 
of disadvantaged clients who said they didn't know if the descriptions 
were "accurate and up to date, " suggest that disadvantaged clients 
may have had less information prior to using OIAS against which to 
judge the information contained in the job descriptions. 

Comments of counselors substantiate and further confirm these 
client ratings. Many counselors participating in the test reported 
that they found it dcfiuitely helpful to obtain as many printouts of 
job descriptions as possible for use in their own daily activity. 
For instance, several attached the descriptions to MDTA referral 
forms rather than having to research and write their own justifica- 
tions. Counselors' opinions of the descriptions was unanimously 
positive and only a few noted minor deficiencies of content. The 
three counselors who indicated deficiencies commented that some of 
the descriptions were not of sufficient depth, that they should be re- 
lated more to the local labor market, and that the amount of informa- 
tion on schooling included in the descriptions was limited. One 
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counselor thought that some of the descriptions were a little too 
long, but all others considered the length satisfactory or very good. 
No counselor . had any suggestions for ^improving style and format of 
the descriptions or on the procedure for getting them. 

Counselors indicated that their clients did not have any trouble 
understanding the descriptions, though one counselor reported that 
sometimes a little explanation was needed. These results are 
consistent with client ratings on ease of reading the descriptions. 

Counselors were asked what effect the terminal had on clients' 
willingness to read the descriptions. They said: '^It gave them 
something 'now' to look at and explore;" "The terminal enhanced the 
comprehension because they read it as it was typed out;" "(The 
terminal) motivated them to read;" and "They would sit there glued 
to it reading. " 

These comments emphasize the attractiveness of the teletype 
terminal usage arid its utility in prompting even slow readers to 
read all that is printed out. This is a result of the pacing effect 
of the word- by-word printout process which enhances the effectiveness 
of the process and tends to insure that the information is read, 
understood, and utilized, especially since the printout is something 
tangible which the client can take with him. These effects are 
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fully consistent with and amplify clients' high ratings on the "fun 
to use" dimension. 

Counselors were asked to rate the overall currency of informa- 
tion contained in the job descriptions. The following table shows 
how counselors rated the information. 

All but one counselor considered the information in the des- 
criptions definitely satisfactory. The one counselor who rated the 
information fairly poor emphasized the geographic factor, saying it 
was not local enough and not sufficiently current for* local needs. 
This counselor remarked about the almost weekly cliange of local 
demands for specific occupations and referred to the sudden but tcni- 
porary effects of labor disputes on local demand in specific occupa- 
.tions. It is apparent that this particular counselor was giving 

TABLE 10 

COUNSELORS' RATINGS OF CURRENCY OF 
INFORMATION IN JOB DESCRIPTIONS 



Very Good 
Fairly Good 
Fairly Poor 
Very Poor 

ERIC 



19% 
75 

6 

0 
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heavy emphasis to immediate employment opportunity rather than 
longer range occupational decision- making. 

In an attempt to discover specifically how the descriptions could 
be improved, counselors were further asked to rate the topics 
covered in the descriptions. In rating each topic, they were cau- 
tioned to ignore any information availability constraints. The 
ratings present probably the best available statement of what 
counselors expect in a good occupational description, as well as 
indicating how satisfactory the OIAS description format proves to be. 
Table 11 presents the results of those ratings. 

Table 11 reveals that the vast majority of counselors found the 
content and topical coverage of the job descriptions satisfactory. 
While they were asked to rate length, they in fact were rating the 
adequacy of information content for each topic on the list. None 
indicated that any topics included were superfluous or unduly em- 
phasized. 

While the majority of counselors rated content of the job des- 
criptions satisfactory on all counts, a portion indicated their desire 
to have more information on several topics. With the exception of 
wages and frings benefits, counselors were almost unanimously satis- 
fied with coverage of all topics concerned with the nature of the job 
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TABLE 11 

COUNSELOKS' RATINGS OF TOPICS OF INFORMATION 
CONTAINED IN OCCUPATIONAL DESCRIPTIONS 



Topic 



Percent of Counselors 
Suggesting: 



More 



Same Less 



Drop 



Nature of Job 
Function 
Job Duties. 

Occupational Specialties 

Working Conditions 

Current EInnployment 

Employers 

Work Environment 

Work Schedules 

Organizat ions 

Wages and Fringe Benefits 

Qualific ations 

Native Qualifications 

Legal Qualifications 

Education, Training, Experience 

Training Sources 

Hiring Channels 

Promotional Ladder 

E mp I oy me nt Prospects 
Demand 
Supply 

Supply/Demand 



0 


d. \J\J /O 


n 


0 


0 


100 - 


0 


0 


0 


ion 


n 


0 


o /o 




0 


0 




1 n n 
1 UU 


U 


u 


8 


92 


0 


0 


0 


100 


0 


0 


0 


100 


0 


0 


23 


77 


0 


0 


31 


69 


0 


0 


23 


77 


0 


0 


38 


62 


0 


0 


31 


69 


0 


0 


31 


69 


0 


0 


31 


69 


0 


0 


31 


69 


0 


0 


15 


85 


0 


0 


15 


85 


0 


0 



and working conditions. Like everyone else, counselors commented 
that they would like to have more localized and more detailed wage 
rate information. 
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Qualifications for employmenl was the area where counselors 
most frequently and consistently indicated a desire for more informa- 
tion. Qualifications, together with wages and dennand provide the 
information which is not only informative for exploratory purposes, 
but useful in determining the appropriateness of an occupational 
choice. (Since completion of the Portland test, additional informa- 
tion on training sources has been developed and included,) 

Counselors' ratings and comments on job descriptions pose 
something of a dilemma. While asking for more information on 
some topics, counselors expressed almost unanimous satisfaction . 
with the overall length of the descriptions. Only one counselor felt 
they were a little too long. Above a third of the counselors asked 
for additional information on a number of items, without compensa- 

. ting reductions in other items. The fact that a large majority of 
counselors found the topical coverage, as well as the overall length 
satisfactory, cautions against wholesale alteration of the descriptions. 
In drawing conclusions from this test, it is important to remember 
that the project resources went primarily to delivery system develop- 

. ment, not to information development. It is actually quite encourag- 
ing that counselors and clients v/ere as satisfied as they were with 
the, information content of the information files during the test. The 
quality of the information content can be expected to rise when the 
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System is implemented permanently and information is maintained 
over a longer time period by means of a more consistent and 
sophisticated information development program. Improvements in 
information on a number of topics have already been made and 
included. 

Other Information Components 

The remaining three information components of OIAS were used 
by only a limited number of clients during the test. Several factors 
contributed to this result. Most clients who used OIAS were satis- 
fied with using the questionnaire, Obtaining a list of occupational 
titles, and getting a few printouts of job descriptions. Only a small 
portion of clients desired more detailed information beyond that con- 
tained in the job descriptions or preferred the other media. It is 
also. true that the Bibliography and Books, the Interview Cassette 
tapes and the Employer Index lacked the appeal of the computer 
terminal for the clients. The- Bibliography and Books and the Enn- 
ployer Index required looking up references, which many clients were 
uninterested in doing. A number of counselors who participated in 
the test were not totally familiar themselves with usage of the Bibli- 
ography r. ad Books and the Employer Index. In the follow-up inter- 
views a number of counselors indicated that they didn^t fully 
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understand the use of the Employer Index until late in the test. 
■ These factors contibuted to the small portion of the client population 
using these three information components. However, sufficient usage 
was made of these components to provide data for some conclusions. 

Bibliography and Books 

' Seven of the seventeen counselors who participated in the test 
indicated that either they didn^t use the Bibliography and Books at 
all, or they used them so little they could not comment on their 
use. The remaining ten counselors confined most of their comments 
to general impressions and reactions. 

Comments included oUch statements as: "I think it was good. 
Wc did a good job of using everything (occupational information 
. publications) we had available in the resource room;" "Fantastic, 
marvelous;" "Bibliography was a tremendous help;" "Adequate. 
Found some people wanted to use it. Generally, the type of client 
we get (is) not interested in reading;" "This is the point where most 
of them dropped out. Only had two who used it ^ and they were both 
college students. General run-of-the-mill client doesn't have long 
enopgh attention span;" "A few people c^%^h£|,ndle these things. 
Most of them are out working now;" "I think the best thing is the 
Occupational Outlook Handbook . I didn^t get a great deal out of the - 
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rest. DonH know if the clients used it to any degree at all/' and, 
"They're time consuming (but) worthwhile if the client really wants 
to dig into something. " 

The counselors who commented in more detail on the Bibli- 
ography and Books all thought this component worth continuing and 
considered the publications included sufficiently comprehensive al- 
though somewhat repetitive. These counselors further thought that 
the bibliography furnished a good reference tool for counselors, even 
if few clients used it. They indicated satisfaction with the format as 
well as the content. 

A total of 35 clients reported that they used the Bibliography 
and Books. Of these clients, six rated this component of OIAS as 
the most helpful. Thirty clients rated a set of characteristics of. 
this component. Table 12 shows these client ratings. 

Table 12 reveals that the vast majority of clients who used the 
Bibliography and Books found them helpful. These ratings must be 
tempered by consideration of the relatively small number of clients 
who used this information source. It is also important to point out 
that clients who used this component are not representative of the 
total client population. As counselor comments indicated, only 
clients who were willing to "dig for information, " and who were suffi-' 
ciently motivated to search independently, used them in the first place. 
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TABLE 12 



EVALUATION OF BIBLIOGRAPHY AND BOOKS BY 
CLIENTS WHO USE THIS COMPONENT 



N on- disadvantaged Disadvantaged 
Clients Clients 



DonH Don't 

Respondents Yes No Know Yes No Know 

(Percent who used this component = 16% of TOTAL Experimental 
Group Clients) 

Accurate and Up 86% 5% 9% 78% 11% 11% 

to Date 

Complete 90 5 5 67 33 0 

Easy to Read 90 10 0 78 22 0 

Fun to Use 91 9 . 0 67 33 0 

Related Job to My Own 
f.jikes and Dislikes, 

Values and Skills 89 11 0 89" 11 0 



Although only a small proportion of clients used this informa- 
tion component, client ratings and counselor comments indicate that 
it is a worthwhile component, and should remain available to that 
select portion of clients who find it helpful and are able to use it. 



68 

Occupational Interview Cassettes 

Only eleven clients reported using the interview cassette 
tapes. At the beginning of the test only 17 occupations were 
available on the tapes, a fact which obviously constrained their 
usage severely, even though the nunnber doubled by the end of the 
test. Sonne of the tapes used were obtained comnnercially, while 
some were developed by project staff. There were some signifi- 
cant differences in length, format, and credibility between the two 
sets of tapes, with those produced by the project being much 
superior; unfortunately, one cannot generally separate comments 
about those from comments about the poorer, commercial tapes. 

Most counselor comments were stated in terms of general 

reactions. The comments of the eleven counselors who did use 

them were as follows: 

"Very limited in number I think. They could be expanded 
(more cassettes) . . . but cassettes used were fine. " 

"They were stolen. I don't know how you control that. 
Thought they were gimmicky and expensive. Suggest 
money (should be) used for better books, pamphlets, and 
give away stuff. Have more printouts available. " 

^'There was a problem of having them out for use. I liked 
to use them because you could hear people talking, but had 
trouble getting kids to use them. They didnH seem to want 
to take the time to use them." 
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''Very limiUKi. Some were loo long for attention. Had 
very few who used theoi, " 

"Some were well done, others bored people. Effective 
for some, " 

"A good idea but not enough fields covered, " 

''Nothing really exciting in this part of System. Found 
client reaction a real either/or situation; either quite inter- 
ested or else didn't want to hear it. Don't know why, " 

"] invited people to use them and there were some people 
who spent some time with them. Others listened for a 
second or two. I do think they are aimed at the high 
school student level, " 

"I listened to a few. See some value in them, but a 
lack of sufficient number," 

"l think they would be good with groups, " 

"Not enough there at the time. I've used them before and 
have had good success with them. Sometimes the cassettes 
work better than just having them read, " 

These comments represent a considerable range of rea'ctions. 
One counselor, already quoted, thought the cassettes should not be 
maintained in the System; another expressed doubts about their 
usefulness in terms of the very small number of clients who used 
them; while the rest of the counselors who commented thought they 
were useful and should be retained in the System, However, all 
had suggestions for improving their use. Expansion of the number 
of cassettes available was a common suggestion. Shortening the 
length of some cassettes was mentioned by a few counselors. 

O 
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(Project cassettes are about half as long as the commercial ones.) 
No counselor reported that clients had any trouble using them, 
although the fact that some were stolen points up a procedural 
problem. Two counselors made specific references to the fact 
that the cassettes were useful with clients who had reading diffi- 
culties. 

A small number of clients rated their experiences using the 
cassettes. These ratings must be considered as only impressionis- 
tic and cannot be considered at all representative even of those who 
used the tape cassettes. Table 13 presents the results of client 
ratings. It is interesting that more disadvantaged persons used 
them, even though the tapes had to be checked out and were not 
openly available near the terminal. 

More thorough testing is needed to determine if any real 
differences in effectiveness exist between disadvantaged and non- 
disadvantaged clients. The only descriptive comments of clients 
were: "told counselor that the tapes had too much irrelevant 
information for economic use of time. The Occupational Outlook 
Handbook was better;" "very interesting;" "more selection;" and 
"wasn't interested. " These ratings and comments present the same 
mixed picture as counselor comments* Unquestionably, there are 
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TAJ3LE 13 

CLIENT EVALUATION OF INTERVIEW 
CASSETTE TAPES 

N on- disadvantaged Disadvantaged 
Clients (N=4) Clients (N=7) 

Respondents Yes No Yes No 

(Percent who used this component: 5%) 

Fun to Use 100% 0 

Easy to Understand 100 0 

Gave a Good Idea of What 

The Work was Really Like 100 0 

Related Job to My Own 
Likes and Dislikes, 

Values and Skills 100 0 



advantages and usefulness of this part of the System, but in individu- 
al counseling situations they are not a high use component. Their 
best usage seems to be with groups. 

Some additional evaluation of the interview cassettes was 
gained when a set was sent to the Tacoma Ghetto Job Information 
project. They were reviewed by Employment Service counseling 
and job placement staff as well as being tested in some high schools. 
Employment Security staff felt the tapes would not be helpful with 
job ready clients, but they did feel that they would be helpful to 



83% 17% 

86 14 

86 14 

71 29 
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young entry workers being referred to trainee positions and with 
clients in general during vocational exploration or orientation 
prior to career counseling. 

When the tapes were used in several live dennonstrations 
before high school audiences, students were "attentive and enthusi- 
astic," The Tacoma Job Ghetto Information staff said students 
iound theni very interesting and helpful and were particularly inter- 
ested in the economic information provided. This additional test of 
the interview cassettes confirms the broad appeal of the cassettes 
in school or other group settings. However, their value in individual 
counseling seems limited to certain kinds of clients at specific points 
in the vocational exploration process. 

Employer Index 

Usage of the Employer Index was also reported by a limited 
number of clients. Three of the eighteen clients, both non- 
disadvantaged and disadvantaged, who used it rated it as the most 
helpful component of the total OIAS System. Counselors, who used 
it themselves reported that it was a helpful source of information 
in their own work. One specific problem with the Employer Index 
in the test was lack of counselor understanding of its use. Another 
difficulty was office policy regarding the use of the Index. Counselors 
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were instructed that the Employer Index was confidential information 
and not to be turned over to clients for their own use. Rather, 
counselors were to use it themselves and relay the appropriate 
information to the client. However, some counselors apparently 
saw an awkwardness in this procedure and allowed clients to use 
it directly to copy down names of potential employers. All coun- 
selors who followed this procedure noted that they explained to 
clients that the listing of potential employers was not evidence of 
any present job openings. 

In spite of limited use, counselors unanimously commented 
that it was a useful tool and worth continuing as a part of the 
System. Those counselors who used it frequently saw it as very 
useful and valuable while counselors who used it only a little tended 
to emphasize that it was difficult to use. Two counselors suggested 
that they would like to have had more-^- specific instruction and 
orientation in use of the Employer Index. 

It is certainly true that use of the Index could be simplified 
by some additional work on design and format. That was recog- 
nized when the Index was created, but was infeasible at thai time. 

No specific evaluation or descriptive comments were obtained 
from clients on the Index. Counselor comments about client use, 
although extremely limited, reveal a definite lack of interest or 
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reluctance of many clients to use it. This is consistent with 
evaluation of usage of the Bibliography and Books and the Interview 
Cassette tapes. Those few clients who definitely wanted the detailed 
information available in these components found them helpful, while 
most clients were satisfied with the information contained in the job 
descriptions, and thus did not use the other information components. 

Client Satisf?^ction with Infer mat ioa 

Results of the Portland test indicate that the information com- 
ponents of OIAS provide satisfactory information content in effective, 
yet varied formats. Very few clients using OIAS fail to obtain the 
information they desire. Most clients are very well satisfied with 
the information contained in the widely popular Job Descriptions, 
which have proven to be the heart of the System, and most saw no 
need to use the additional components. However, for about 10 pei*- 
cent of the clients with somewhat differing information demands and 
different media preferences, the Bibliography and Books, the Inter- 
view Cassettes, or the Employer Index provide the additional means 
to obtain the particular information to fill their special needs. The 
System would be functional without these three components, but per- 
haps ten percent of the clients would miss what is, for them, the 
most helpful part of the System. 
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CHAPTER VIII 

DIFFERENCES BETWEEN CONTROL AND EXPERIMENTAL 
GROUPS IN SOURCES OF INFORMATION 
AND TIME REQUIRED 

While the experimental groups were using the OIAS files, 
another (control) group of clients was getting information from 
counselors in the usual way. The differences in information 
sources used by control and experimental groups reveal further 
differences between the traditional counseling process and the 
counseling process when a systematized information access process 
is utilized, ^ 

■ Sources of Information Used With Control Group 

Counselors were asked to name the publications and sources 
of occupational information they used with each client in the control 
group. Most counselors listed more than one source of information. 
The following table ranks these sources according to their frequency 
of use. 
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TABLE 14 

INFORMATION. SOURCES AND THEIR FREQUENCY OF USE 
WITH CONTROL GROUP CLIENTS 



Information Source Frequency of Use 



Dictionary of Occupational Titles 


31% 


Occupational Outlook Handbook 


31 


College Catalogs 


31 


General Aptitude Test Battery 




Testing and Interpretation 


27 


Apprenticeship Information Center 




Publications (Resource Room)'*' 


16 


Job Bank Book 


15 


Pamphlets and Miscellaneous Publications 


15 


Interest Checklist 


10 


Civil Service Publications 


10 


Current Labor Market Information 


5 


Miscellaneous Publications''"'*' 


13 



"'Refers to use of occupational materials assembled and 
available for client use at the Youth Opportunity Center, A variety 
of pamphlets, leaflets, and reference materials are available for 
client use in one room. 

'■''•'This category includes "Encyclopedia of Careers" . (cited 
twice), Stanford Interest test (cited twice), "Portland Manpower 
Survey," "Mapping Your Education," "College Placement Annual" 
and private school source, material (each cited once). 

Counselors reported using two information sources, on the 
average, with each client. The preceding table shows that coun- 
selors rely on a limited number of information sources. The first 
four sources listed are consistently the most widely used, .but none 



ERIC 



77 



of them is used more than a third of the time. Of all sources 
listed, the Occupational Outlook Handbook is the only publication 
which provides compreheng^ive inforhiation for specific occupations. 
Of the four most frequently used sources, it is the only source 
that provides any supply and demand information, and even the 
Occupational Outlook Handbook must often be tempered by local 
labor market information in order to adequately determine local 
opportunity. The other sources are limited to specialized and limited 
aspects of occupational information. 

A few counselors reported that they used no published sources, 
but provided occupational information from their own personal know- 
ledge or from telephone calls to schools, other agencies, and 
en;iployers. Counselors working with clients in experimental' 'groups 
were not queried on sources of information utilized, but it can be 
assumed that they communicate their personal knowledge and make 
personal contacts while working with clients whenever it is appro- 
priate. Thus, the use of personal knowledge can be considered a 
constant factor for both control and experimental group counseling 
activity. The differences in information sources between control 
and experimental groups consist of differences between the published 
information sources listed in Table 14 for the control group and the 
systematized information files contained in OIAS. Clients in the 
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control f»roup wer^e not queried, so there is no way of determining 
what information those clients actually read, investigated, or utilized 
except by inference from the information sources counselors re- 
ported using. Consequently, it is not possible to determine whether 
clients in the experimental groups sought more information than 
clients in the control group. 

In the experimental groups who used OIAS, itemization of 
particular books was reported in the counselor evaluation of usage 
of the Bibliography and Books. This component received limited 
use by clients because the OIAS descriptions are the first choice 
information source of OLAS users, but, when it was used, the 
Occupational Outlook Handbook was the most frequently used book 
with. OIAS as well as in the traditional counseling process. 

The job descriptions were by far the most widely used OIAS 
information component. The content of the descriptions is some- 
what parallel to the coverage of topics in the Occupational Outlook 
Handbook . Both sources provided comprehensive coverage of a 
wide range of specific occupations, and do so in a consistent for- 
mat. However, the information in the job descriptions is far more 
succinct, geographically localized, and up-to-date. Clients also 
rate them easier to use. When given a choice, clients chose the 
OIAS descriptions first, then turned to the Occupational Outlook 
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1 land book > The general appeal and value of the occupational des- 
cinptions is further substantiated by the fact that 90 percent of all 
users rated them as the most helpful of the OIAS information files- 
(not counting the questionnaire). 

"Orange Flowered Box" File 

One source of information used with both control and experi- 
mental groups at the Youth Opportunity Center (YOG) deserves 
special mention. YOG has a specific resource room containing 
occupational information. Within the room is a set of files which 
their staff refer to as the "Orange Flowered Box" file, which con- 
tains files (in boxes printed with orange flowers) of pamphlets and 
articles on specific occupations. This is a valuable information 
source for clients and is widely used. 

The "Orange Flowered Box" file was listed on the Bibliography 
in all three offices* It was physically available only in the YOG 
office, so clients from the Adult Opportunity Genter (AOG) or the 
Goncentrated. Employment Program (GEP) would have to travel to 
the YOG office to use it. This situation provided an opportunity to 
discover whether people would travel across town to use such a 
source. No counselors or clients reported that anyone from AOG 
or GEP did so. Thus, if such files of printed information are to be 

O 
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developed, they must be conveniently located, presumably in each 
office where clients might want to use them. 

Efficiency of Information Presentation 

Efficiency is a vital element in information presentation, and 
access time is an essential consideration in determining the value 
of an information system. OIAS presented occupational information 
in a more unified, integrated, and straightforward manner than the 
traditional counseling process, and it also delivered more information 
in the same amount of time. 

Estimates of the amount of information access time 'were 
gathered from counselors and experimental group clients. Coun- 
selors were asked to estimate the amount of time it would have 
taken to obtain equally adequate information to use with each client 
in the experimental groups if OIAS had not been available. The 
median time reported for both non-disadvantaged and disadvantaged 
clients was 60 minutes. That is, counselors thought it would have 
taken them an hour to find the information their clients got from 
OIAS. In a few instances, counselors indicated they would never 
have been able to obtain the same amount of information. 
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Clients were asked to estimate the amount of time they spent 
using the System. The median estimated time was 40 minutes for 
non-disadvantaged clients and 38 minutes for disadvantaged clients. 
Thus, within 40 minutes clients were able to obtain information 
which, on the average, would have taken a counselor 60 minutes 
to assemble, and additional time to deliver. 

The median time spent by clients in using the System can be 
considered the average time required to complete .the questionnaire, 
obtain a list of occupational titles, and obtain one or more specific 
job descriptions. The majority of clients did not seek additional 
information from the other components' of the System. Usage of 
those components would require more client time, but, from the 
viewpoint of staffing, time spent by counselors is the primary con- 
cern. Counselors generally agreed that independent client use of 
the Bibliography and Books and the Interview Cassettes would pose 
no problem for those clients capable and desirous of using thetfi. . 

Counselors' general comments during follow-up interviews 

indicated OIAS usage generally did not save great amounts of time, 

> 

when compared with the time usually spent on information delivery, 
but made more and better quality information available within that 
time. Because these comments were not generalized and made after 
the test was completed, they reflect the total time required to use 
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OlAS under normal use conditions better than the time logged on 
the various steps of the test. The lack of adequate scheduling of 
terminal usage, the inefficient, temporary location of the terminals 
in the offices, which forced counselors to run back and forth from 
their work areas, and the fact that counselors provided all the 
technical assistance to clients using the System all consumed coun- 
selor time during the test. Thus, if OIAS were installed in these 
offices on a permanent basis, adequate scheduling and development 
of procedure, Nvhich would integrate OLAS into ongoing office 
operation, would likely result in a reduction in the 40- minute 
estimate of counselors' staff time spent per client. 
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CHAPTER IX 

COMPARISON BETWEEN COMPUTER AND 
CARD- SORT VERSIONS 

Two alternative processes for obtaining a list of occupational 
titles and job descriptions were used in the Portland test. One is 
the coniputer version of OIAS; the other is a manual, card- sort 
version. Both use the same QUEST questionnaire; the difference 
.between the two versions lies in the procedures for obtaining the 
list of occupations. Whereas clients in the connputer group typed 
their responses into a teletype terminal, clients in the card- sort 
group used a long needle to sort a stack of cards. Each card con- 
tains a job description for a specific occupation. When the client 
finishes sorting the cards, those remaining in the stack are the 
same occupations printed out on the terminal in the computer version. 

Utilization of these two methods for delivering occupational 
information provides a comparison of their relative merits. The 
major dimensions considered are: the relative effectiveness and 
ease of usage of each version, the amount of lime required of 
counselors and clients to use each system, client understanding of 
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. the processes, and the extent to which clients were able to utilize 
the information obtained from each. Analysis of data points out 
advantages and disadvantages for each version. 

While the card- sort has..all the job descriptions of occupations 
remaining in a client's stack immediately available, the computer 
version requires the user to request the printouts of job descriptions 
from the terminal one at a time. There are two important factors 
involved here. First, the card- sort system delivers more information 
(descriptions for every occupation on his list) than the user is 
interested in reading. Secondly, the attractiveness of the terminal 
is not matched by any feature of the card- sort version. This 
directly affects the client's willingness to read the descriptions. 
Clients are more likely to read carefully the descriptions in the 
computer version. (Additionally, Job descriptions in the computer 
version can be easily updated; whereas the individual cards in the 
card- sort version must be reprinted and replaced individually.) 

Counselors described their clients' reactions to the computer 
version as positive. This was true for disadvantaged as well as 
non- disadvantaged clients. Counselors said that clients were: 

^'Enthusiastic. " 

'^Enjoyed it. " 

''Was intrigued by the terminal." (disadvantaged) 

O ' . ... 
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was fascinated; stimulated; we related better as a result 
of sliaring the experience, even though we had rapport at the 
outset, " (disadvantaged) 

"This man was very enthusiastic. It caused him to drop 
his smooth, 'not a care in the world' attitude, and we really 
got down to the fact that he was quite a depressed man, 
with very good reason, " 

"Very good- -he had no trouble in handling this on his own-- 
counselor checked every so often and he was progressing 
very well. " (disadvantaged) 

The same reaction occurred with the card- sort, when used by 

non-disadvantaged clients. But, disadvantaged clients were generally 

\ 

unimpressed vvith the card-sort. The reasons for lack of interest 
in the card- sort by disadvantaged seems to be the failure of this . 
medium to motivate these users. 

The great attractiveness of the terminal in client use was most 
apparent in the reading of job descriptions. Counselor comments 
strongly indicated that clients read all the job descriptions which 
they ordered from the terminal. The word-by-word printing process 
paced clients' reading of these descriptions, and this is a major, 
frequently noted advantage. The attractiveness of the terminal and 
the immediacy of the information in the computer version provides 
assurance that clients will use it independently. 

The card- sort produced much less consistent results. When 
counselors were asked if clients usually read most of the descriptions 
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on the cards that remain on their list, 60 percent of the counselors 
stated that clients didn't, 33 percent reported that clients read most 
remaining descriptions, while only one counselor stated that clients 
read them in detail. Typically, the client would skim through the 
cards remaininj^ in his deck, loo!: at titles and select and read only 
those of interest to him. Clients usually sorted the cards at the 
counselor's desk, and that fact had some influence on the clients' 
persistence in using the card- sort. This fact also explains the 
divergence of counselor opinion as to whether clients read the 
descriptions. Tiu..-;^. a (disadvantage with the card- sort version is 
thf^ upp'oriainty whether clients will read the descriptions sufficiently 
carefully to obtain information essential to their occupational decision- 
making. 

In the follow-up interviews, counselors were asked to compare 
the two versions and give their general reactions. Ten of the seven- 
teen counselors participating in the test preferred the computer 
version and thought it was more effective than the card- sort. Only 
one counselor expressed preference for the card- sort version, saying 
she personally liked the card- sort better, but the computer was in 
fact a more attractive, effective version for her clients. The re- 
mainder remarked that each version has some advantages and disad- 
vantages, but most of them considered the computer version more 
efficient. 
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In Iheso comments the attractiveness and efficiency of the 
terminal were the characteristics most often mentioned. Frequent 
comments about computer down-'time revealed this as the major 
disadvantage of the computer version during the test. In some 
instances, the computer wen!' down after the client had started 
entering his QUEST respo-ises. Most of the inconvenience centered 
on clients not being able to use the terminal immediately and some- 
times having to return another day to complete the process. This 
is, of course, a fault of the computer system being used, rather 
than the OIAS program, but it does highlight the necessity of com- 
puter reliability. 

Ease of usage was another advantage of the computer version 
reported by counselors. Even those who expressed a mixed reaction* 
in comparing the alternative versions thought the terminal easier 
for clients to understand and use. 

The main advantage cited for the card-sort was its greater, 
effectiveness in showing the client what was happening in the 
sorting process and how each response affected the outcome. The 
most frequently mentioned disadvantage of the card- sort was its 
cumbersomeness. Cards tended, to stick together and sometimes 
didn't drop out properly. 
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Returning to clients' evaluations, their assessments of the 
Job Descriptions show some differences which are also a function 
of the version of the System used. Clients definitely thought the 
descriptions were easier and more fun to read when the..computer 
version was used. The impact of the computer medium is evident 
in the clients ratings of the popular occupational descriptions. 



TABLE 15 

CLIENTS' EVALUATION OF 
OCCUPATIONAL DESCRIPTIONS 





Definitely 
Yes 


Yes 


No 


Don't 
Know 


COMPUTER GROUP 










Accurate and Up to Date 


33% 


57% 


5% 


5% 


Complete 


35 


50 


10 


5 


Easy to Read 


41 


55 


4 


0 


Fun to Use 


45 


54 


1 


0 


Related to My Own Likes and 
Dislikes, Values and Skills 


32 


57 


11 


0 


CARD- SORT GROUP 










Accurate and Up to Date 


21 


59 


3 


17 


Complete 


13 


64 


8 


15 


Easy to Read 


28 


67 


5 


0 


Fun to Use 


21 


72 


7 


0 


Related to My Own Likes and 
Dislikes, Values and Skills 


24 


63 


13 


0 
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Clients gave a consistently higher rating for the descriptions de- 
livered via computer, including rating it higher on accuracy, com- 
pleteness, and relevance. These differences primarily reflect 
differences in the attractiveness of the two versions, because the 
informational content was very similar, with the single exception 
that the descriptions delivered by card- sort were not as highly 
localized or as frequently updated. 

Clients' ratings of how they liked information they received 
from OIAS further substantiates this difference. Again, the 
responses refer more to the relative appeal of the processes than 
the content of information. Table 16 reflects these ratings. 

.TABLE 16 

CLIENTS' RATINGS OF HOW WELL THEY 
LIKED INFORMATION THEY RECEIVED 
FROM OIAS 







Computer 
Group 


Card- Sort 
Group 


Like It 


Very Much 


47% 


31% 


Like IX 




47 


61 


JDislike 


It 


5 


7 


Dislike 


It Very Much 


1 


1 
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Over no pei'cenl of the clients liked the infoi'mation, regardless 
of the method they used to access it, but, again a substantially 
higher proportion of those using the computer version expressed 
strong liking for the information. On the negative side, there are 
only slight differences between the two groups in the percentage of 
negative responses, with the exception of the "fun to use" category 
in Table 15. 

Another index of ease of usage were clients' reports of trouble 
using OIAS. Eighteen percent of clients in the computer group re- . 
ported having some trouble using OIAS, while only 9 percent of the 
card- sort group had difficulty. While there appears to be a signifi- 
cant difference between the two groups on this point, these ratings 
are inconclusive. Computer down-time was already mentioned as a 
recurrent problem during the test and certainly is reflected in 
clients' reports of difficulty. It is not an inherent difficulty in 
the design of the OIAS system, however. Because of the frequency 
of this mechanical problem, it is difficult to separate it from diffi- 
culties experienced by clients in understanding and following the • 
instructions. 

A similar problem is encountered in the card- sort. The 
tendency of the cards to stick together caused frequent sorting 
problems, as reported by many of the participating counselors. 



erJc 



91 . 

This again is a slraighlforward mechanical problem, which can be 
corrected by using different card stock. Thus, with both versions, 
the amount of trouble with procedures and instructions in the process 
itself is partly obscured by mechanical problems, but comments of 
clients help identify some of the problems. Those who had trouble 
using the System were asknd to describe their difficulty. A few 
reported some problems with the questionnaire and a number of 
disadvantaged clients cited reading and understanding the instructions 
as the specific problem. 

The remaining comments pertained primarily to the mechanical 
and procedural difficulties noted above. The only two comments 
identifiably linked with the card- sort version referred to the cards 
sticking together. Computer down-time was specifically noted in 
several instances and a number of minor problems were cited for 
the computer version, most commonly by disadvantaged clients. The 
following comments of clients describe such problems: "I forgot to 
type in HELLO. I also had some minor problems figuring out what 
key to punch;" "Locating keys . . . not a typist;" and "I forgot 
to put in the period." These are relatively minor problems which 
can be solved quite simply with routine technical assistance at the 
terminal site. 

O 

ERIC 



J 



92 

Usefulness of Information 

Differences in the usefulness of infornnation provided by the 
two versions can be gleaned fronn clients. Results show that both 
versions effectively assist clients in nnaking job plans. There are 
only slight differences in the usefulness of infornnation received 
fronn each version. Sixty-eight percent of clients using the connputer 
version reported that they were helped, while 64 percent of the 
clients who used the card- sort indicated OIAS helped thenn. Overall, 
a strong nnajority of clients who used the System indicated they 
received definite help. 

However, the relatively high percentage of clients who reported 
not being helped needs explanation. Clients tended to answer this 
question in terms of whether they received sufficient information to 
make a definite choice or take a step in the occupational decision- 
making process. This is further clarified and substantiated by client 
responses to the question whether they liked the information they 
received from OIAS as shown in Table 16, 

The ratings on "liking" reflect the personal relevancy and use- 
fulness of the information as well as the attractiveness of the process 
of obtaining it. All these factors are necessarily subjective and the 
responses on the part of clients are impressionistic. However, the 
responses of clients who reported that OIAS helped them make job 
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plans indicate that useful infornnation was communicated either in 
the process of filling out the questionnaire and getting a. list or 
from the information obtained from the information components. 
Usefulness and helpfulness are closely related dimensions. Of 
course, the ultimate criterion of the usefulness of information is 
whether it significantly affects the labor market behavior of persons 
who use the System, but that kind of evaluation is beyond the scope 
of the present study. 

A very high proportion of clients reported that they liked the 
information and found it useful; although it may not have resulted in 
a definite and immediate step in making job plans. The lower 
proportion of clients who indicated they were helped in making their 
job plans as compared to the clients who reported that they liked 
the information, reflects the differences between clients who- made 
definite decisions and those who didn't but still found the information 
sufficiently relevant to consider. A greater proportion of clients who 
used the computer version reported liking the information very much; 
however, clients overwhelmingly liked the information from both 
versions. 
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Summary 

Both versions of OIAS are effective and efficient means of 
providing occupational information, although both have some minor 
mechanical problems. Time is saved in finding pertinent occupa- 
tional information. The computer is appreciably more attractive to 
clients and preferred by counselors; while the card- sort requires 
more technical assistance and direction. Solution of the minor 
mechanical problems and development of a procedure^for providing 
routine technical assistance will further enhance the effectiveness 
of both versions. 
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CHAPTER X 

DIFFERENCES BETWEEN NON-DISADVANTAGED AND 
DISADVANTAGED CLIENTS IN USAGE OF OIAS 

Evaluation of differences between non-disadvantaged and disad- 
vantaged clients in OIAS usage was an important dimension of 
investigation. At issue was the ability of disadvantaged clients to 
operate and benefit from the System. (Previous investigation had 
shown the System's value, but had not differentiated these groups.) 

Care was taken to insure that both disadvantaged and non- 
disadvantaged clients were included in the control and experimental 
groups, and that they received comparable treatment. Counselors 
recorded each client's disadvantaged status along with other client 
characteristics, but otherwise made no distinction between disadvan- 
taged and non-disadvantaged. The grouping of clients as disadvanta^ged 
and non-disadvantagfid was done at the stages of data compilation and 
analysis only, and thus, presumably did not influence counselor re- 
sponses. Clients were classified by counselors as disadvantaged or 
non-disadvantaged on the basis of Federal criteria currently used 
by the State Employment Division. 
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In general, the OIAS System proved effective with both groups, 
but within the broad category of disadvantaged, it is necessary to 
identify the severely disadvantaged. In this test, the greatest 
problems emerged among those disadvantaged 'persons who had severe 
educational and reading deficiencies. Those who had serious trouble 
operating the System are those who, according to responses, had 
definite reading difficulties. Counselor comments in follow-up 
interviews made reference to the inappropr iatene.-js of OIAS for 
persons with extremely poor or no reading skills. 

Unfortunately, the te^it did not produce precise criteria for 
identifying clients for whom OIAS is inappropriate. Test data, 
both reading skill level and GATE, were collected for all clients 
participating in the OIAS evaluation, when such test results '.vere 
available as a result of normal counseling procedure. However, 
the differences in tests present insuperable problems of analysis. 
The Concentrated Employment Program administered the Wide 
Range Achievement Test to almost everyone entering the program, 
but this instrument was not used in either of the other two offices. 
The YOG and AOC use reading tests infrequently, and when they do, 
they use forms of the ABLE. Both use the GATE widely, but at 
the Concentrated Employment Program (CEP) the GATE is. used 
less frequently. When the GATE is used at CEP, it is generally 
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with clients who appear capable o! doing sufficiently" well on it.. The 
NAT.B is rarely used. To compound these problems there were few 
severely disadvantaged clients with severe reading problems within 
the total disadvantaged group, (This may have reflected in the 
choice of clients for the test; though counselors reported no such 
selective activity. ) Thus, no aptitude or reading level prerequisites 
for use of OIAS can be set, though it is known that persons with 
extremel;> poor reading skill do hav*? trouble operating OIAS alone. 

A more general analysis of differences between the two broader 
groups of disadvantaged and non- disadvantaged is traced through 
client and counselor responses to major aspects of the total system. 

The overall effectiveness and usability of OIAS for disadvan- 
taged and non-disadvantaged clients depends upon the clients' and 
the counselors' purpose for using OIAS. The crucial questions, 
then, are whether it helps clients move toward or reach their 
objectives, to what extent it helps them make job plans, and 
whether the client finds the process and content of information 
presented satisfactorily and relevant. Differences along these 
dimensions point out the different patterns of use and the relative 
effectiveness of OIAS for disadvantaged and non-disadvantaged clients. 

In the evaluation, each client who used OIAS was asked to 
report his or her purpose for using OIAS, and there was a distinctly 
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different pattern of reasons for using OIAS given by disadvantaged 
and non- d isadvantaged client's. Table 18 shows the pattern of 
responses. 

Many disadvantaged clients clainn to be .job seeking, and 
thus seem to be more decided and definite about their occupational 
choices, whereas the majority of non- disadvantaged clients used 
OIAS to help decide, what occupation to follow. 

This significant difference raises a crucial question which was 
not directly addressed in this evaluation. What amount and quality 
of occupational information was possessed by clients prior to OIAS 
usage? On the basis of general chairacteristic differences between 
disadvantaged and non- disadvantaged populations, one would suppose 
that the quantity and quality of occupational information possessed by 
disadvantaged persons is significantly less than the information 
possessed by non-disadvantaged persons. The lower level of formal 
education, the less amount of job skills, the greater amount of un- 
employment and sporadic employment and job discrimination are 
general characteristics of the disadvantaged population which decrease 
exposure to knowledge and information about occupations. Perhaps 
for this reason, perhaps for more basic cultural reasons, there 
appears to be a significantly different pattern and logic to the 
decision- making process between disadvantaged and non-disadvantaged 
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clients. Aware of the complexity and range of occupational choice 
facing them, non-disadvantaged clients reflect a high degree of 
indecision and lack of certainty. In addition, long-term planning 
and dec ision- making is more characteristic of non-disadvantaged 
persons. This is consistent with the high proportion who reported 
they were undecided about what occupation to follow. 

The client's evaluation of the usefulness of OlAS is obviously 
made in context of the purpose for which the client used it, 
whether exploration or confirmation. Table 17 summarizes these 
responses of clients regarding the extent to which OIAS helped 
them make job plans. 

Table 17 shows definite differences in responses between 
disadvantaged and non-disadvantaged clients. A smaller proportion 
of disadvantaged indicated they were helped in each category of 
purpose. Interestingly, the le'ast differences between client groups 
existed when they indicated they were undecided and thus interested 
in exploration. Here the System was nearly as effective with dis- 
advantaged as non-disadvantaged persons. However, for those 
non-disadvantaged clients who indicated they need information about 
some specific occupation they had in m.ind, 90 percent reported 
receiving help. Disadvantaged clients looking for a job were the 
group reporting being least helped. . 
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TABLE 17 

CLIENTS' RATINGS OF HELPFULNESS OF OIAS 
IN MAKING JOB PLANS ACCORDING TO 
CLIENT PURPOSE FOR USING IT 



100 



Reasons Used 



Total 
Clients 



Defi- 
nitely 
Yes 



Yes 



No 



Defi- 
nitely 
Not 



N on- Disadvantaged Total=78 

A. Looking for a Job 20% 19% 

B. Had some future occu- 
pation in mind that I 
needed to find out 

more about 13 20 

C. Undecided about what 

occupation to follow 67 15 



62% 19% ■ 0% 



70 10 



54 21 10 



33% 36% 7% 



Disadvantaged Total==102 

A. Looking for a job 41% 24% 

B. Had some future occu- 
pation in mind that I 
needed to find out 

more about 31 18 

C. Undecided about what 

occupation to follow 27 18 



44 32 



43 28 11 



The fact that many more disadvantaged were job seeking and 

over twice as many disadvantaged clients as non- disadvantaged 
clients had some future occupation in mind that they wanted to find 
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out moi'c about is also consistent with the difference in patterns 
of purpose. These responses tend to substantiate the conclusion 
that disadvantaged persons have less prior occupational information 
upon which to make judgments, so they decide upon an occupation 
within a narrowly perceived range of alternatives. It is also 
consistent with their shorter time horizon, and their predisposition 
to think in terms of jobs, not "careers," 

Elucidation of these differences is augmented further by tho 
ratings of counselors on the purpose of using information with 
clients. The response patterns shown in Table 18 are significant, 
not because of the differences between client groups on this dimen- 
sion, but because of the differences as they relate to the preceding 
discussion. These data reveal a very significant difference between 
client and counselor perceptions of the purpose of using OIAS and 
the information presented through it. 

The overall differences in ratings for disadvantaged and non- 
disadvantaged clients show that counselors saw. exploration as the 
major purpose for using occupational information for both kinds of 
clients. They saw slightly fewer disadvantaged clients at a point of 
decision, but twice as many as having reached a decision previously 
and needing to confirm that decision through use of OIAS. This 
counselor's view, that more disadvantaged used OIAS to confirm a 

ERLC 



102 



TABLE 18 

PURPOSE OF USING OCCUPATIONAL 
INFORMATION WITH CLIENTS AS - 
RATED BY COUNSELORS 



N on- Disadvantaged Disadvantaged 
Purpose Clients Clients 

Exploration 68% 57% • 

Decision 18 15 

Confirmation 14 28 

previous decision, further substantiates the preceding observation 
that disadvantaged clients expressed a greater degree of certainty 
of choice than did non-disadvantaged clients prior to OIAS usage. 

Of perhaps equal innportance, however, are the differences 
between the perceptions of the purpose of using occupational infornna- 
tion held by disadvantaged clients and those held by their counselors. 
While the data in Table 17 and Table 18 are not strictly comparable, 
there are some relationships which express some very pointed 
differences. Only 27 percent of the disadvantaged clients indicated 
that they were using OIAS because they were undecided about what 
occupation to follow, whereas counselors saw exploration as the main 
purpose of using occupational information with disadvantaged clients 
in 57 percent of the cases reported. At the very least, there is 
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considerable disparity between counselors and these clients about 
the clients' occupational planning status. These counselor ratings 
represent counselor judgnnents about where a client is in the occu- 
pational decision- making process. In a sense it is also an assess- 
ment by the counselor of the realism of the client's choices and 
decision, and the disparity may express the likelihood that the 
disadvantaged client, has made prior choices or decisions based upon 
a narrowly perceived range of alternatives which in turn was based 
upon a definite lack of adequate occupational information. 

In contrast to the differences of counselor and disadvantaged 
client ratings, there are very close similarities between counselor 
and non-disadvantaged client ratings. Sixty- seven percent of the 
non-disadvantaged clients indicated that their purpose in using OIAS 
was to help decide what occupation to follow. Counselors rated 
the purpose of information use with 68 percent of non-disadvantaged 
clients for the purpose of exploration. Since there is a high level 
of consistency between these two ratings, this finding further 
sharpens the difference found with disadvantaged clients. 

More non- disadvantaged than disadvantaged clients saw OIAS 
providing at least some help in job planning, but it is important to 
note that the client who was seeking confirmation of a prior choice 
and who received no additional helpful information would tend to 
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I'espond in the "no" category. This point leads to another, broader 
factor. Since clients were included for test purposes who might 
not be referred to OIAS if it were available on a permanent basis, 
the proportion of those who were riot helped by OIAS is probably 
somewhat exaggerated. Client desire and counselor judgment of 
the appropriateness of System usage on an ongoing basis would 
certainly reduce the proportion of negative responses. 

in order to better understand, it is helpful to examine the 
tenor of client descriptive comments reporting why they were not 
helped by OIAS. The comments of disadvantaged clients were: 
"Because it didn't say anything about what I want;" "Jobs unrealistic 
for my qualifications;" and "It did not have the job I was looking 
for." Negative comments of non-disadvantaged clients ranged from 
"Not sure;" "Unsure at this time;" and "Still undecided;" to "Not 
enout^jh different jobs." All other descriptive comments for both 
groups related how they were helped. 

The preceding negative comments cited are also consistent 
with the differences in pattern in the decision- making process 
between disadvantaged and non-disadvantaged persons hypothesized 
above. Two of the three comments of disadvantaged persons indi- 
cated that OIAS didn't help them because it failed to confirm a 
prior choice. In contrast, the factor of uncertainty dominated the 
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responses of the non- disadvantaged clients. The positive comments 
of both groups, however, were highly similar, indicating that both 
non- disadvantaged and disadvantaged clients were helped in a very 
similar way. 

Whereas the intensity of their negative comments differ, there 
is much similarity in the positive comments made by both, groups 
of clients. Disadvantaged clients said they were helped in different 
ways: 

''By telling me basically what I should know and where to 
obtain training. 

^'l didn^t know there were so many fields related to the one 
I had in mind to follow." 

"Made me think. " 

"By supplying me with information that was vital for my 
own consideration. " 

"Gave me some ideas." 

"Gave me some other occupations to consider." 

"By explaining in detail the work involved and the job 
opportunities offered. " 

Nondisadvaataged clients commented that OIAS; 
"Gave me some ideas where to look for a job. " 
"Eliminated some possibilities and confirmed others." 
"Told me where to get training I need. " 
"Gave me some new ideas." 
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"Plelped by telling me more about what I had to know, what 
kind of training and education 1 had to have. " 

A related aspect of the difference was expressed in response 
of clients when queried whether they liked or disliked the informa- 
tion they received from OIAS. Only one out of 79 non- disadvantaged 
clients who responded said they disliked it, while 11 percent of the 
disadvantaged clients indicated they disliked it. These findings point 
out that, while a third of the clients reported that OIAS didnH help 
them make job plans, they overwhelmingly liked the information 
received. Thus, client perception of whether OIAS helped appears 
to include other factors. If OIAS usage decisively resulted in a 
clear step in the occupational decision- making process, it was 
viewed as helpful. If usage produced results which the client didn't 
like, a job they wouldn't want for instance, they responded with a 
negative rating. There were a number of descriptive comments- 
reported by disadvantaged clients. These comments included: 

"(Questionnaire) needs more questions." 

"It, didn't give me any specific job." 

"Unrealistic; did not take my age into consideration. " 

"The answer it came but with; I didn't like that kind of work." 

"The only problem was I didn't get auto mechanic. " 
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"I'm a woman, they had jobs I thought more suited to a man." 

"They did not have anything on cook. " 
The majority of these comments focus on a specific job which the 
client had previously chosen, or a specific result desired by the 
client. This type of response is characteristic of what disadvantaged 
clients hoped to obtain from using OIAS. There was a tendency for 
these clients to have a definite expectation about the results of OIAS. 
When this didn't occur, they were dissatisfied. This accounts 
partially for the- lower amount of satisfaction and help disadvantaged 
clients reported obtaining from using OIAS. 

Another crucial que^jtion of comparison between the two 
groups of clients was whether disadvantaged clients had more 
trouble using OIAS. A specific question in the client evaluation 
questionnaire was directed to this point. Slightly more disadvantaged 
than non-disadvantaged persons reported that they had some difficulty 
using the System, but these differences are not significant, though 
the types of problems differ. Over half of the comments of non- 
disadvantaged clients specified a problem of a mechanical nature, 
while over half of the problems experienced by disadvantaged per- 
sons were difficulties with reading and comprehending either the 
questionnaire or the instruction for terminal use. Two disadvantaged 
clients specifically said reading was the problem. This is to be 
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expected. However, responses and comments of both groups of 
clients clearly show that OIAS can be used easily and effectively 
by most disadvantaged as well as non-disadvantaged clients. 

The findings and analysis of the test of OIAS reveal a dis^ 
tincl difference in the decision- making processes between disadvan- 
taged and non-disadvantaged clients. This is an important finding 
for counseling as well as for information delivery systems. Despite 
these differences, OIAS helped both by expanding the range of pos- 
sible occupations by providing information through which the client 
can eliminate some occupations and by providing highly detailed 
specific information. Thus, OIAS is sufficiently broad in scope to 
effectively relate the same occupational information to the needs of 
both disadvantaged and non-disadvantaged clients. 



TABLE 19 



CLIENT RESPONSES TO WHETHER THEY 
HAD ANY TROUBLE USING OIAS 



Non- Disadvantaged 
Clj'dnts 



Disadvantaged 
Clients 



YES 
NO 



12% 
88 



15% 
85 



109 



APPENDIX 
METHODOLOGY 

This report is based on a field test of the Occupational Infor- 
mation Access System in the Counseling units of three Oregon 
Employment Division offices in Portland, the Concentrated Employ- 
ment Program, Youth Opportunity Center, and the Adult Opportunity 
Center. Counseling supervisors from those offices participated 
with the project monitor for the agency, in the planning and opera- 
tion of the test. 

Evaluation design for a project such as OIAS is difficult 
because it is infeasible to systematically evaluate many, let alone 
all, of the plausible uses of the System. Furthermore, rigorous 
evaluation is often achieved at the expense of creative suggestion 
and innovative application, both of which are highly valuable in a 
developmental project. Another problem is ihat users and evalualors 
are influenced by many non- system features of the test: adequacy 
of training, other workload, management attitudes, etc. Substantial 
.. training and frequent opportunity for communication between project 

staff and local office staff were utilized as ways of handling the 
latter type of problem. 
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Counseling- staff turnover was a problem, particularly in the 
Concentrated Employment Program. Staff reduction was occurring 
at the time the OlAS test began. During the course of the lest and 
evaluation, two of the four CEP. team counselors resigned and another 
was transferred. Of these, two were unavailable for follow-up inter- 
views. In each of the other two offices one counselor had left and 
was unavailable for the final follow-up interview. 

Evaluation Design . The evaluation design followed traditional 
experimental- -coiitrol group methods. The evaluation was intended 



to assess: 



Us^^bilitv: 



Performance: 



Efficiency: 



e.g. Can disadvantaged clients operate the 
System themselves? How easy is the System 
for clients to use? 

e.g. To what extent is use of information in- 
creased? Is the information helpful to clients?- 
What effect does the information have on client 
decisions ? 

e.g. Staff time required. Time saved. 



Suggestions: e. g. Process or content improvements suggested 
by staff or clients. 

Each of the three Employment Service test offices was to 

select 100 clients who were in the process of making or changing 

occupational choice and in need of occupational information. These 

clients were to be randomly assigned to three groups: 

- - experimental group: computerized system 
--experimental group: card- sort system 
--control group 
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('I'hc methods of study population selection and group assign- 
ment varied with office procedures, ) ; 
Three data sources were planned: 
Counselor report: 

• Client characteristics 

Type of use: exploration, decision, confirmation 

Use mode: self, technical assistance, counselor assistance 

Components used: QUEST, DESC, BIBL- BOOKS, 

Cassettes, Employer Index 
Time cost: time spent, potential time saving 
Performance: effect on decisions, resultant actions 
Suggestions: application, content 

Client report: 
ease of use 

helpfulness of information 
use made of information 
suggestions for improvement 

Counselor records: 

references to occupational information 
number of counseling interviews/case 

retention and successful closing, e.g, incidence of placement 
Data from the first two sources were collected by the coun- 
selors and compiled by the project staff. Data from counselor 
records, which were to.be compiled by the agency following the 
test, were omitted for lack of time and difficulty in accessing the 
records. 

In all, 17 counselors and 267 clients (including 66 in 
a control group) participated in the Portland test. Random 
assignment of clients to the control group and the two 
O experimental groups and the fact that each counselor 
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worked with clients in each of the groups provided control for 
client and counselor differences. During the test no distinction 
was made between disadvantaged and non- disadvantaged clients. 
Counselors simply completed a basic data sheet for each client 
on which they checked the client's disadvantaged status along with 
other characteristics. The breakdown of experimental group clients 
into disadvantaged and non-disadvantaged was done at the stage of 
data compilation and analysis. While a few counselors admitted 
some minor screening of clients, it was judged that this activity 
did not alter the results of the test. 

Individual counselors' case loads were not specialized; such 
differences in client characteristics as did exist were only a func- 
tion of the differences in clientele served by the different offices. 
Characteristics of clients participating in the test are shown in 
Appendix Table 1. : 

Data for the evaluation were collected by means of three 
instruments: a questionnaire completed by the counselor following 
each client^s use of the System, a questionnaire completed by the 
client (experimental groups only), and an interview conducted with 
each (Counselor following completion of the test. Copies of all 
these questionnaires appear at the end of this appendix. 
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Some clienla and counselors did not complete every item , in 
the evaluation questionnaires. Thus, calculations are made on ihc 
basis of the number who responded on each item, rather than on the 
total number of individuals participating. 

APPENDIX TABLE 1 

NUMBER AND. CHARACTERISTICS OF CLIENTS 
PARTICIPATING IN PORTLAND TEST 



Control Experimental Groups 

Client Characteristics Group Non-Disadvantaged Disadvantaged 



Disadvantaged Status 



Disadvantaged 33 -- 115 

Non-Disadvantaged 33 86 



Age 

Under 22 33 37 61 

22-54 33 47 52 

55+ 0 2 2 

Highest School Grade 
Completed 

0-7 0 0 .2 

8-11 . 14 13 44 

12 , 40 33 52 

Over 12 12 40 17 

Clients Using Computer 

Version, Total: 107 -- 44 63 

Clients Using Card- Sort 

Version, Total: 94 -- 42 52 
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Planning and Administration , Planning the Portland test in- 
volved a number of interested parties. While the test outline was 
prepared by the project director, modifications were made based 
on suggestions by both USES national office and state agency 
personnel. Written comments and approval were received from' 
Manpower Administration national office staff, including counseling 
and testing,' labor market information, and occupational analysis 
divisions; both verbal and written comments were received from 
the state counseling supervisor and state supervisor of technical 
development and analysis. Meetings with state central office staff 
and Portland area staff (distrid srpervisor, office managers, 
counseling supervisors, counselors) also contributed to test design, 
including decisions about components to be tested and procedures 
to be used in local offices. 

The State Chief of Research served as the Oregon agency's 
chief contact person during the Portland test as he had throughout 
the project; the district supervisor and office managers participated 
in the decision to carry out the test, the location of the terminals, 
and other matters affecting the total office. The State Counseling 
Supervisor indicated that he would not take a highly active role 
in the test out of concern that the test appear to be something imposed 
on local offices by the central office, and schedule conflicts pre- 
vented his attendance at some meetings; nevertheless he, along with 
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other state office pei'sonnel, received periodic information from 
the agency's project monitor. 

Whereas state agency personnel at all levels participated in 
planning, the administration of the Portland test was delegated by 
the agency to a small group composed of the Chief of Research 
representing the central office, the local office counseling superb- 
visors representing the three test sites, and the project director. 
That group met on numerous occasions to clarify procedures, plan 
staff training, recommend locations for teletype terminals, etc. 

Staff training in the use of the System and in completion of 
the evaluation instruments was conducted July 16, 19, and 20, 1971 
by the project staff, with one session held for the counselors in each 
of the test offices. Day-to day supervision of the test was in the 
hands of the local office counseling supervisors. The project staff 
also made several visits to each of the offices during the test to 
answer questions, clarify procedures, check on progress, etc. 
On-site visits were also made by various state-office staff and tv/o 
national- office staff. Some of these visits led to further improvements 
in the administration of the test and clarification of issues for the report 

Counselor workload placed constraints on the test just as it 
does for counseling services generally. Counselors were asked to 
use and evaluate OIAS without special adjustments in workloads. 
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As noted elsewhere in the report, test procedures required coun- 
selors to spend more time with OIAS than they would have spent 
had they been tree to utilize it as they thought best. Despite 
the added work created by the OIAS test, counselors were highly 
cooperative with the conduct of the test. 



COUNSELOR EVALUATION OF O.I.A.S. USAGE 

Client 



Counselor^ 

Office 

Client. Charecteristics (pv^iilPble from ES-511) 

1. Highest School r-rcde: (check one) 

0-7 ( ) 

\ ] 

Over 12 ( ) 

2. Age: (check one) 

Under 22 ( ) 
( ) 

55^ ( ) 

3. Disadvantaged: (check one) 

Yes ( ) Reason: 

No ( ) 

h. Aptitude Test Scores (if administered) GATE ( ) NATB ( ) 

G= V= N=: S= ■ It Q= F= 



5. Reading Level (if tested) 

Stanford - Primary II ( ), Int. II ( ), Adv. ( ) 
AELE - Level I ( )> Level II ( ) 
WRAT (CEP) ( ) 

Notes: 

6. Which version of the system did you use with this client (check one): 

Computer ( ) 
Card-sort ( ) 

Neither (control group) ( ) (Answer Question 7, then skip to Question lO) 

7. (FOR CONTROL GROUP ONLY) 

Please describe briefly what occupational information you gave and what 
sources you used. Please identify materials used, organizations contacted, 
etc. (You may use the back of this page to continue your answer.) 
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2. 

8. Client Ability to Use the System (for Computer and Card Sort Groups) 

Please indicate who did each of the following steps. Circle the most appro- 
priate code: 

1. client aioae 

2. client with oaly technical assistance from the counselor 

3- client with counseling and interpretatio n from the counselor 
i+. counselor, with client watching or helping 
5- counselor alone (using results indirectly in counseling) 
About how much of your time did this step take? 
QUESTIONNAIRE AMD LIST (if not used, write "not used"): 

Staff 

Step 1: Filling out the questionnaire 12 3^5 Time: 

— Counselor review responses with client-- 

Step 2: Reading and understanding the instructions 

for using the terminal or card-sort (last Staff 

page of the Questionnaire) 12 3^5 Time:_ 



Step 3: Typing answers in the terminal; or, sorting Staff 

the cards 12 3^5 Time: 

Staff" 

Step k: Reviewing the list of occupational titles 12 3^5 Time:^ 
— Counselor discuss the list with client-- 

Step 5: (Optional): 

"CHANGE," "START OVER,'' "VTOT NOT" (circle Staff 

the ones used, if any) 12 3^5 .Time:_ 



Your Comments on Proc'^dure: 



Client Reaction: 



OCCUPATIONAL DESCRIPTIONS (if not used, write "not used"): 

Step L: Getting the Description from the terminal Staff 
or card deck 12 3^5 Time: 



3* 

Step 2: Reading the Description from the terminal 

or card 1 2 3 h 3 Time:_ 

Step 3: Interpreting the Description 1 2 3 ^ 3 Time:^ 

How. many Descriptions were taken (if known)? 

Comments on style and format (readability, organization, etc.)s 



Comments on content (topics .covered, accuracy, etc. Especially note other 
information requested by the client.): 



BIBLIOGRAFHY AJTO BOOKS (if not used, write ''not used"): 

Step 1: Looking up the Bibliography she^t tii the 

Bibliography Notebook 12 3^5 Time: 

Step 2: Selecting the book to read 12 3^5 Time: 

Step 3: Reading the book or report 12 3^5 Time: 

Step k: Interpreting book or report information 12 3^5 Time: 

Wiiich Books were used? 



Comments : 



INTERVIEW CASSETTES TAPES (if not used, write "not used"): 



step 1: 


Finding the Cassette 


1 


2 


3 


k 


5 


Time: 


Step 2: 


Operating the Cassette Flayer 


1 


2 


3 


k 


5 


Time ; 


Step 3: 


Listening to the Cassette 


1 


2 


3 




5 


Time: 


Step it: 


Interpreting the Cassette 


1 


2 


3 


k 


5 


Time: 



o 
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k. 

EMPLOYER INDEX (used by counselor) (if not used, write "not used"): 

Staff Time : 

Comments on Procedure: (identifying Occupational Specialties, using 
Employer Index print-out, finding other necessary information, presenting 
information to client.) 



9. (Computer and Card-sort groups only) 



Su^ipose you did not have OIAS available. Jt'iease estimate how long it would 

have taken you to get eaually adequate information to use with your client? 

~ (estimate) 

Comments: 



10. Hov/ woLilr! you characterize the purpose for using occupational information 
with this client? (check one) 

Exploration : To facilitate exploration of occupational opportunities 

and the factors to be considered in making a choice. ( ) 

Decision : To get information on which to base a decision ( ) 

Confirmation ; To get information to validate a tentative decision 

already arrived at ( ) 

11. Please describe briefly how the use of occupational information effected 
client decisions. 

Rate: Information was: Highly influential ( ) Somewhat influential ( ) 

Not influential ( ) 

12. Can you identify any actions by the client that are attributable (partly or 
completely) to use of this information? Please describe briefly and try to 
indicate the influence of OIAS (if OIAS was used.) 
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--to be completed by clients 



Il&me 

The Occupationel lafoi^nir tion Access System (OIAS) is vei^y ne\u and we ax^e 
trying it out to see how much it helpc people. We want to know what you think 
of it. 

1. When you used it: (check one) 

a. Were you looking for a job? { ) 

b. Or, did you have some future occupations in 

mind that you needed to find out more about ( ) 

c. Or, were you undecided about what occupation 

to follow? ( ) 

2. Did OIAS help you make your job plans? 

Definitely Yes ( ) Yes ( ) No ( ) Definitei^^ No ( ) 

How? 



3, How many different times did you use the system? 

h. Altogether, about how much time did you spend using the system? 



5. Did you have any trouble using OIAS? 

Yes ( ) No ( ) 
If so, whtb kind oi trouble? 



6. How did you like the information you got from OIAS? 

a) Like it very much ( ) c) Disliked it ( ) 

b) Like it ( ) d) Disliked it very much ( ) 

What did you especially like or dislike about it? 
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7. Which parts of- tne system did you use: (check the parts you used] 
Questionnaire end list of ,iob titles 
Job Descriptions 

Biblicgrpphy Notebook , books, end other written material 
Interview Cassette Tapes 
Employer Index (from your counselor) 
Which one was most helpful? (check one) Which one was least helpful? 



Why; 



Questionnaire end list 


( ) 


Questionnaire and list 


Job Descriptions 




Job Descriptions 


Bibliography and Books 




Bibliography and Books 


Interview Cassette Tapes 


( ) 


Interview Cassette Tapes 


Employer Index 


( ) 


Employer Index 






Why? 



( ) 

( 

( ) 
( ) 
( ) 



Yes 
Definitely 



8a. Job Descriptions (if you used them) 
Accurate and up-to-date ( ) 



Complete (covered all 
important topics) 

Easy to read 



Fun to use 

Related the Job to my own 
likes 'end dislikes, 
values and skills 



( ) 
( ) 
( ) 

( ) 



Coniments: 



Yes 



Definitely 
No Not 



()()() 



( ) 
( ) 



( ) ( ) 
( ) ( ) 



()()() 
()()() 



8b. Bibliography find Books (if you used them) 



Accurate aad up-to-date 

Complete (covered all 
important topics) 



( ) 
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Easy to read 
Fun to use 

Related the Job to my own likes and 
dislikes, values and 
skills 



( ) 

( ) 
( ) 

( ) 



( ) ( ) ( ) 

()()() 

()()() 
()()() 

()()() 



Don't 
Know 



( ) 
( ) 



( ) 
( ) 



3. 

Yes Definitely Don't .. 

Definitely Yes No Not Know 

8c. The Interview Cassette Tapes (if they were used) 

Fun to use ( ) ()()() 

Easy to understand () ()() () 

Gave a good idea what the 

work is really like ( ) ()()() ( ) 

Related the job to my own 
likes and dislikes, 

values and skills () ()() () 

Which tepe(s) did you use? 

Comments: 

6d, The Questionnaire (if you used it) 

Easy to read ( ) ()()() 

Fun to use ( ) ()()(() 

Asked the right questions () ()()() 

Related jobs to my own 
likes and dislikes, 

values and skills ( ) ()()() 

Did the questionnaire give you sorae new job titles that you would con- 
aider for future work? 

Definitely Yes ( ) No ( ) 

Yes ( ) Definitely No ( ) 

Comments: 



9. If you need job information later, would you want to use OIAS again? 
Yes ( ) No ( ) 
Why? 



10a. What information would you use that OIAS didn't have? 
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10b, Can you think of other things that would make OIAS better? 



11. Would you like to see OIAS kept at the Employment Division? 

Definitely Yes ( } No ( ) 

Yes ( ) Definitely No ( ) 

12. Other comments: . . 



NOTES TO INTERVIEWERS 



Taping: Request, but don't ' insist upon taping. 



Candor: Encourage frank responses. We want candid responses 

both negative and supportive. 



Confidentiality: Assure thenn that they will not be quoted by name. Request 

pernaission to excerpt the tape for an audio report. 

Notes: - Make brief notes (sunnmaries) about 

questionnaire, so we'll have an idea 
response without going to the tape, 
quotable responses. 

Periodically, write the tape position beside a question to 
help in locating that response. 



responses on the 
of the counselor's 
Note especially 



Redundancy: If you feel you have already answered a particular question, 

just say so and we'll go on. 

Key Questions: You may need to cut down the length of the interview 
because a counselor is uncooperative or is not well 
informed about the system-^Csome used it only a few 
times,) In that case, you may want to use just the 
key questions v/hich are marked with asterisks. 



! 
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FINAL 

COUNSELOR EVALUATION OF 
O.I.A.S. USAGE^* 

Counselor 

Office 

Number Clients Who 
^ Used OLAS ^ 

Interviewer^ 

GENERAL 

^General Comments: What is your general reaction to OIAS? 

^•To what extent did OIAS make it possible for you to use information in 
counseling that you would not have used otherwise? 

How much time did it save you, say in a week? 
^Which part of the system do you like best? Why? 
*Which part of the system do you like least? Why? 

*V/e have noticed that usage has dropped off recently. Why do you think 
that is? 

^Are there parts of OIAS you would like to see kept in your office 
permanently? 

*Do you think the Portland test provided a fair evaluation of OIAS? Are 
there things the evaluation didn't take into account? 
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APPROPRIATE CLIENTS: 

*Are there special things about your own case load that affected the test 
of CIAS? (Note special case loads , e.g. youth at YOC) 

*Did you find CIAS more useful with one kind of client than another? 
• Which? 

Were there any types of clients that it was especially bad with? 



Were those differences equally true for both the card- sort and computer 
versions? 



CARD- SORT vs COMPUTER: 

^How would you compare the manual {card- sort) and computer versions? 
Is one much easier for clients to understand and use? Which? 



Is one much more attractive to clients? 



Is one much more effective with clients? How? 



Is one a more convenient resource for your own use (work with clients, 
MDTA referral justification, etc. )? Did you use it much as your 
own reference source? How? 



(If .not already mentioned): Was computer down time a problem for you? 
Are there other things about, either system which made it difficult to use? 



I have two questions specifically about the card-sort system: 
Do clients usually read most of the Descriptions on the cards that remain 
on their list? 
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•We*ve thought of shortening the material on the cards to just three 
or four brief statements about the occupation and perhaps adding some 
graphics. We, could then provide a printout of the Descriptions^ like 
, those from the computer, for further reference. What do you think 
of that idea? Would clients look up the Descriptions? 

QUESTIONNAIRE and LIST 

^General Reaction: What is your general reaction to the QUEST 
Questionnaire and list? 



^Readability: How would you rate the reading difficult of the questionnaire 
for your clients? 

Are there particular questions you found hard for your clients to 
understand? 



Are there particular questions you found i nappropriate for your clients? 



*Pd like you to quickly look through the questionnaire and tell nie, for 
each question, whether you would suggest keeping it as it is, changing 
it, or dropping it. [NOTE: Tryj to get suggestions as to what kind's 
of changes are desirable. ] ' 



Keep As 
Is 



Change 
(How?) 



Drop 



Physical Limitations : 

1 . Lifting 

2. Impaired Vision 

3 . Deafness 



( ) 
( ) 
( ) 



Location: 

4. Region 

5. City Size 

6. Working Conditions 

7. Sex 



( ) 

( ) 



( ) 

( ) 

< ) 

( ) 



(Some people have suggested di^opping the sex question, because a 
client's response might be based on stereotypes about the sexes. 
Would you recommend keeping or dropping that question? Why?) 
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Education and Training: 
8. Education 



( ) 



( ) 



( ) 



Aptitudea: 

9. Working with Hands 

10. Eye for Accuracy 

11. Ability v/ith Words 

12. Working with Numbers 

13. Catching on to Things 
(The interest questions are based on the data-people-1 
cation of the DOT. Do you think they are a good basis for clients 

' expressions of interest? 



( 
( 

.( 
( 

• ( 

hings classifi- 



Interests: 

Working with Things: 

14. Precision Work 

15. Operating Equipment 

16. Handling Materials 

Working with Information: 

17. Interpreting Facts 

18. Organizing and Using 

Information 

19. Copying, Sorting, Putting 

things together 



( ) 
( ) 
( ) 



( ) 
( ) 



{ ) 



( ) 
( ) 
( ) 



( ) 
( ) 



( ) 



( ) 
( ) 
( ) 



( ) 
( ) 



( ) 



Working with People: 

20, Counseling 
" 21. " Discussing, Bargaining 

22. Teaching, Supervising 

23. Selling, Persuading 

24. Assisting 



( ) 

( ) 

( > 

( ) 

( ) 



( )• 

( ) 

( ) 

( ) 

( ) 



Earnings: 

25. Monthly Wage 



( ) 



( ) 



Are there other questions which should be asked? What? 



{ > 
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*If you were free to use the questionnaire any way you wanted to in your 
counseling, how would you use it? 

[Interviewer: use the following codes when appropriate:] 



1, Client alone 

2, Client with only technical assistance from the counselor 

3, Client with counseling and interpretation from the counselor 
[Note: clarify whether counselor considers his input necessary 

or desirable] 

4, Counselor , with client watching or helping 

5, Counselor alone {with results indirectly in counseling,) 

Step 1: Filling out the questionnaire 1 2 3 4 5 

Step 2: Reading and understanding the instructions for using 

the terminal or card- sort (last page of the Questionnaire 

1 2 3 4 5 



Step 3,: Typing answers in the terminal; or, 

sorting the cards 12 3 4 5 



Step 4: Reviewing the list of occupational titles 1 2 3 4 5 

Step 5: (Oiptional): 

"CHANGE/' "START OVER," "WHY NOT" 1 2 3 4 5 

(Circle the ones used frequently) 

During the test we said a counselor should review the 
client's questionnaire with him before getting a list. 
How necessary is that? 

We also said the counselor should discuss the list with 
his client. How important would you say that is? 



LIST: 



What about the length of .the Lists? Were there any particular problems 
associated with that? 



Did you find the ^'C luster" or occupational family format of the list 
helpful, or would it have been better to use DOT codes? 

^Were there particular things about the lists that were confusing to clients? 
What? 



*Could you explain the reason why certain occupations appeared on the list 
and others did not? [Know how to use WHY NOT? 
ATTR coding accurate?] 

*Do you think there is a danger of clients taking the list too seriously? 

*Is this a greater danger with the pomputer version? 

*How do you feel about independent client use of the QUEST? Hov/ 
important is it for a counselor to be involved? [Summarize 

statement; circle one: - 

essential desirable it depends undesirable] . 

DESCRIPTIONS: 

^General Reaction: Generally, what is your opinion of the Descriptions?- 
Do you have any comments on the procedures used for getting Descriptions 



Should clients be allowed to get Descriptions independently, or is counselor 
interpretation necessary? If so, why? 
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READABILITY: 

Did your clients have trouble understanding the Descriptions? 

What effect did the terminal have on client's willingness to read the 
Descriptions? Why, do you think? 

Can you suggest improvements in style or format of the Descriptions? 

'f^Hovv would you rate the currency of the information? (circle one:) 
Very good Fairly good Fairly poor . Very po^r 

*What do you think of the overall length of the Descriptions? 
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Here is a typical Description, and here is a list of topics usually covered. How 

could the space be better allocated? (Ignore information availability constraints). 

L ength 

More Same Less Drop 

NATURE OF JOB 

Function: ( ) ( ) ( ) ( ) ' 

tTob Duties: - ( ) ( ) ( ) ( ) . 

Occupational Specialities: ( ) ( ) ( ) ( ) 

WORKING CONDITIONS 

Current Employment: ( ) ( ) ( ) ( ) 

Employers: ( ) ( ) ( ) ( ) 

Work Environment: ( ) ( ) ( ) ( ) 

Work Schedule: ( ) ( ) ( ) ( ) 

Organizations: ( ) ( ) . ( ) ( ) 

Wages & Fringe Benefits: ( } { ) ( ) ( ) 

QUALIFICATIONS 

Native Qualifications: ( ) ( ) ( ) ' ( ) 

Legal Qualifications: ( ) ( ) ( ) • ( ) 

Education, Training, 

Expeinence: ( ) ( ) ( ) ( ) 

Training Sources: ( ) ( ) ( ) ( ) 

Hiring Channels: ( ) { ) ( ) ( ) 

Promotional Ladder: ( ) { ) ( ) ( ) 

EMPLOYMENT PROSPECTS 

Demand: ( ) ( ) { ) ( ) 

Supply: ( ) ( ) ( ) { ) 

Supply /Demand: ( ) ( ) ( ) ( ) 
OTHEli^TOPICS 
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*Do you remember any particular things being wrong or ambiguous in 
any of the Descriptions, things that we should check into? 

BIBLIOGRAPHY and BOOKS: 

^General Reactions: How do you feel about the Bibliography and Books? 

Can clients use it independently? 

Is the Bibliography worth continuing? 

How is it best used? 

What do you think of the number of publications in the Bibliographies? 

What do you think of the particular publications included? Are there others 
you would Like to see included? 

What other information about the publications would be helpful? 

^Suggestions? 

CASSETTES 

^General Reactions: 
Did clients have trouble operating the cassettes themselves? 

Is this a useful part of the system?' 

How are the cassettes best used? 

How is the length of the cassette interviews? 
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Are there other questions which should be asked on the tapes? 

Is there other information we should add to the tapes? 

Are there particular occupations you would like to have tapes for? 

* Suggestions? 

EMPLOYER INDEX 
^General Reactions: 
Is this component worth continuing? 

Comments on using the Employer Index: 

1. Identifying Occupational Specialties (Typing EMPLY occupational code); 

{getting DOT codes and titles) , 

■ 

2. . Did you have any trouble using Employer Index Print- outs? 

3. Did you have trouble finding other necessary information 

(addresses phone numbers, etc.?) 

4. How did you present the information to your client? 
Why wasn't the Employer Index used more, do you think? 

^Suggestions? 
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CONCLUDING COMMENTS 

What do you think should be done next with OLAS? 



Are there other kinds of information you would like to have readily 
available? 



Do you have any final things you want to say before we finish? 
Requests: 



We may decide to produce a taped report, using excerpts from some of these 
evaluation interviews. Is it all right with you if we use part of this tape in 
that way. We would not use your name. 

Yes, you may use the tape { ), - 

No, I'd rather you didn^t quote me { ), 
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